One of the challenges in adopting any methodology in an enterprise is in ascertaining the degree to which it is actually used. Although this sounds simplistic enough it brings a key issue out in the open. Even if an organization has converged on a single methodology to use there are subjective and often acrimonious debates on the extent to which a defined process adheres to the key concepts – either fundamental concepts that outline the methodology or the control objectives that have been bolted on to this methodology. It is not enough to have a methodology preamble announcing that this methodology meets so and so concept or follows this philosophy. A process mature organization needs to go behind the rhetoric (and I am not trying to slam this sort of collateral that sells the methodology) and actually give specific mapping between the process elements and the concepts and or control objectives that are promised to be met with this process.

This data-mapping will not only help in auditing your processes for compliance to your business or regulatory objectives but will also serve as an indisputable baseline to have those debates on what kind of process does the project team really need for an upcoming project. If you know where you are now, it is easier to steer where you want to go. Categorization of your process elements is therefore a simple but powerful way to keep measuring if your process is on track to become what you expected it to be or of you need to make modifications so that you can control its flavor at process authoring stage. Off course another benefit in this approach is that when you are ready to share your process with a peer network they can give their feedback so that you can adjust the mapping of your process elements to the methodology concepts and control objectives and validate this mapping.

There are many examples of this sort of categorization that can be seen in the processes available at the IRIS Process Central Sandbox. You are welcome to try out these processes and provide your comments.

I am pleased to announce the availability of IRIS Process Central Sandbox. This is an online process community to promote discussions on process improvements through collaborative input. Its goal is to bring together process modelers, process consumers, and other stakeholders to freely share information on existing processes and create new process mash-ups.

To facilitate these discussions, we have included several processes on sandbox including MSF for CMMI, OpenUP, and Agile. Where possible, we have provided variety of formats for each process as well as enactment templates.

You can participate by providing your feedback through blogs, discussions boards or modifying the content of the processes. We are eager to get your feedback and will make every effort to ensure that IRIS Process Central Sandbox becomes a valuable asset to you and your organization. Click IRIS Process Central Sandbox to access the sandbox.

IRIS Process Central Sandbox demonstrates how IRIS Process Central which is included as part of IRIS Process Author can help an organization improve their process tailoring in a collaborative environment.

It is our strong belief that through awareness and involvement of process community, there is a great chance of increasing adaptation of processes across the organization. We feel that IRIS Process Central is an important step towards this goal.

We are looking forward to the community participation in IRIS Process Central Sandbox or any suggestions regarding IRIS Process Central product.

In my view IBM is struggling to align their products and practices with the advancement in the state of around software development process automation and collaborative development environment.

In this post I am going to focus on their release of Rational Method Composer 7.2 (RMC 7.2) and specifically point out why it is difficult for me to be convinced that they have done a root cause analysis of their previous failures (with RPW or earlier versions of RMC) or have recognized the direction where software process automation is headed.

When one thinks of a product release from IBM you would obviously expect a high level of maturity, sophistication and thought leadership. IBM is universally well regarded and respected for pushing concepts like SOA and software-as-service. However, when you look at how the Rational team has implemented the tooling around creating and tailoring processes, one wonders why that thought leadership is not being put in practice here. By choosing to keep RMC 7.2 as a thick client application the rational team seems to be stuck in a time warp. I simply cannot imagine a scenario where an enterprise is told that its process assets (which in many cases have the “secret sauce” of project success) is encased in a few laptops. So I was bit surprised that even while acknowledging this problem of scalability and multi-methodology blending, the rational team has chosen to resolve it by continuing to stick to the flawed design decision of using SCM to check-in/check-out process plug-ins. But now, to make matters even more difficult, process authors (often non-programmers), are required to use the eclipse IDE environment to derive some scalability benefits. This is simply unbelievable. First of all, this still does not solve the problem of scalability (see why). Secondly, SCM is a code repository specially built for handling code and not process assets. Finally there is little logic in making process authoring so complicated that it requires a team of consultants to even setup the multi-layer environment for process authoring. Moreover in this day and age when even videos are being edited online asking a process team to work in this client-only disconnected mode is simply irresponsible. I did notice that the rational team did take note of the overwhelming negative feedback on usability of RMC and I welcome that. As in IBM partner and while exhibiting at the RSDC in June 07 we heard lot of this negative feedback. As you read this please keep in mind that if you don’t have a business need for blended methodology processes (and RUP is all you need) or if you are fine with having one or two process authors, RMC does meet your needs. In fact we have a handful of customers who are in such a situation and would simply like to move their RUP process into a collaborative development environment such as VSTS using Content Bridge for VSTS. I hope to see some improvements in the UI with RMC 7.2 and will share what I find out.

Software development governance is concerned with establishing standards and control mechanisms to enable practioners in a development organization to carry out their roles and responsibilities while building or maintaining software development programs. A software development governance framework should not only facilitate the creation of processes that may act as a guide for project teams based on accumulated best practices from experience, but it should also serve to ensure that essential procedures have been followed. Therefore any serious attempt to establish a sustainable software development governance framework needs to build or reinforce these three key pillars: establishing and declaring software development processes, automating the enforcement of governance processes, institutionalization of compliance best practices so that teams are empowered to take corrective action earlier rather than discovering problems later where penalties may result.

When we started researching the current literature and tooling to help us understand the current state of art we were unable to find a single resource that elaborated how to transition from theory to practice. This has made us start work on a whitepaper that would dig deeper while giving an implementation model for software development governance. In this paper we aim to describe how it is possible to deploy a software governance framework including its automated compliance using a collaborative development environment. We use Microsoft’s Visual Studio Team System (VSTS) to illustrate and give a practical recipe for achieving governance automation. After talking about setting up a governance process architecture we guide you through the steps needed to achieve a comprehensive, continuous detection and validation of process enactment and out-of-compliance events. To that end, we also plan to include an implementation reference model which can act as a base for your own specific implementation. By installing an automated governance infrastructure, preparing for internal and external audits requires fewer people and less effort.

I will surely blog about this again when the whitepaper is publicly available.

What is a process architecture ? My last post talked about a project manager tailoring a process or flexibly enacting a tailored process in a project. In this post I would like to zoom out and present how a typical process architecture looks like. Later I would share the litmus test that shows if your process architecture has been created correctly.

In a typical enterprise organization, the top layer in a process architecture consists of process elements. These are elements like roles, work products, activities that go on to constitute a process. These elements could belong to one or more commercial frameworks or could belong to different methodologies or frameworks. These could also include constitutional elements which the organization needs to define and declare in order to meet a internal or external control objective as part of regulatory or quality compliance.

The next layer would consist of process components that are fragments or clusters of related activities. These process components make it convenient to concentrate on specific aspects of a process. Depending on the business or compliancy objectives of the organization you could craft these components to achieve a certain result or describe a desired workflow. One or more components could be linked to create a larger component and eventually an end-to-end process.

Below the process element and process component layer you will have the process layer. This is where you will have the various sub-organizations or departments start crafting their processes in tandem with the software process engineering group. You would not only use elements from more than one packages but in all probability you will be working with other process authors in different time zones or geographical locations. This process layer also ensures that rather than relying on one monolithic process pushed by a vendor or consultant, teams can create processes drawing from different methodologies and process frameworks.

The next layer would make it convenient for the project management office to tailor the process for a category of projects. This domain layer makes it convenient for you to tailor processes for a technology or business domain. Some common examples I have seen in this layer are tailored processes for web applications or those for mobile applications. This domain specific tailoring makes it convenient for project managers to easily search and use processes in real projects. This takes us to the last and final layer, the engagement layer.

The engagement layer consists of processes that are tailored for a specific project. The project manager does not have to go any higher than the domain layer to look for a process that she can specialize for a project that is about to be initiated. Project managers already have their plate full with all the project start-up tasks and they don’t want any overhead associated with searching for and tailoring a process for their project. They want a tactical intuitive way to tailor a process for a project. If they end up spending more time than needed to prepare a process it is likely that this would end up costing more than the benefits accrued. So the litmus test for a successful process architecture is the time taken by a project manager to tailor and operationalize a process in a real project. The lower this transaction cost of tailoring and moving a process to enactment the higher marks you get for a well managed process architecture.

Almost no discussion on managing processes is complete without the use of the word “tailoring”. It is one of those words that although is understood commonly at a generic level starts giving trouble when applied without enough explanation in process development scenarios. I will attempt to describe how I understand “tailoring” in context of process models. I will provide this explanation using a practical scenario of applying a process in real projects.

A process is of little use unless it is expected to add value resulting in successful delivery of real life projects. Every endeavour or project, be it a software project, science experiment or manufacturing an article of clothing, has a certain process it follows. For us in the knowledge intensive domain of software development, successful project delivery is made tougher due to changing or late arising requirements coupled with involvement of skilled knowledge workers. My earlier post talks a bit more on these challenges. Over time, an organization would accumulate a large number of lessons learned as well as patterns and practices that work consistently well across projects. They will also have a collection of anti-processes (think anti-patterns) which they know don’t work. Some organizations would use a tool such as IRIS Process Author to structure their body of process knowledge for easy reuse.

Let us now add a temporal element relating to a point of time when an expected endeavour/project becomes known. At this time, you would study the characteristics of the known execution environment and try to modify your existing body of process knowledge to come up with a process model that best suits this expected project environment. As much as you can, you would also account for the known unknowns in your process model. Based on different situations, this would mean defining things like - a set of activities and their work breakdown structure or workflow, a set of work products that are created and or consumed in the process, the process roles that are staffed by practioners, the pre-post conditions that might be true within the process and so on. You may also defined the narrative guidance that would be read by the practioners in the project. This tailoring of the available process body of knowledge will result in bespoke processes suitable for a specific project execution environment. This process model would best describe the process framework that is expected to result in a successful execution of the project.

Now we will add the second temporal element relating to a point of time when the project is initiated. At this time you would use the tailored process to create instances of process definition elements as project work items. For example this would mean a project manager creating instances of a change control workflow as and when a change work item is added. I would call this instantiation of process definition as flexible process enactment. Keep in mind that the project manager can also add additional work items that are not defined in the process model. However, these changes to the process model could not have been anticipated as the time of process tailoring before the project is initiated. Some would argue that even this flexible process enactment is a “tailoring” of sorts since the process is being instantiated based on the unfolding project (and in some cases new work items are added to the process model). Personally, I have no problems with that as long as there is clarity on the context – tailoring before project initiation and tailoring after project initiation - is understood.

The key point I want to underline is that you would not leave process tailoring to be done after the project has started. Rather you would attempt to tailor your process, using the lessons learned from the past and the known unknowns of the project, before the project is initiated. Once the project starts you would flexibly instantiate your process elements as project work items driven by the circumstances of the unfolding project. Off course you will also add the unknown unknowns to your process. However these by their very definition would not have been known before project startup so could not have been modeled ahead of time.

In my experience process authoring for software development processes is a highly collaborative activity. It is unnatural to expect a solo effort to come up with a process that could be successfully consumed by a wide team of practioners. The process infrastructure also needs to scale well. Reusing existing process components not only reduces significant overhead but also makes the task of coming up with customized processes agile. Any tool that attempts to support process modeling needs to address these two objectives. Based on the evidence I have noticed, the EPF Composer (and RMC from IBM) fails in this regard.

From the information available, it appears to me that there are significant problems being faced by the Open UP team as they try to come up with a process model. The fundamental issue stems from the fact the EPF Composer is designed as a single-user client tool over-relying on SCM to check-in/check-out complicated process plugins. In my opinion it is unrealistic to manage processes in this manner. You basically end up choosing one of two undesirable options. The first option is to break up a process into multiple constituent plugins that are managed separately in an SCM. The second option is to put everything under one library that is checked in. Here is an on-going EPF-dev discussion thread on this topic. This is a thread questioning the rationale for having a overblown library with every sort of plugin in it.

Most enterprise teams will probably face similar problems when using Rational Method Composer (RMC). If the OpenUP team is having difficulties dealing with the limitations of the tool when trying to add a single (DSDM) plugin to the base OpenUP methodology, imagine the magnitude of problem in an enterprise environment when a blended methodology process architecture consists of a number of methods, regulatory and compliancy standards. Moreover, in that environment there will be a large number of subject matter experts who would likely want to collaborate as they contribute to various aspects of the processes. With the difficulties I am seeing with the usage of EPF Composer within the EPF team one can imagine the problems being intensified with scale.

Providing a server side support for process management is critical for a scalable process architecture and collaborative development of processes. This was a fundamental business objective for the Osellus engineering team that came up with a robust database-backed enterprise architecture for IRIS Process Author. Since every process asset resides in a database, it is easy for IRIS Process Author users to discover, tailor and export processes as needed. All this is done without resorting to complicated check-in/check-out rules associated with handling processes managed as files.

Finally, some would say that it is ironic that the very team that is promoting the “eclipse way” is actually building products that seemingly takes one down the path of single client development rather than team-based development or worse still providing only limited visibility into source process content.

In my experience I have seldom come across large enterprise organizations that mandate the use of a specific IDE while putting a blanket ban on any other IDE. Even within the same sub-organization, different development teams may have own their favorite IDEs. Moreover, the decision to use a particular IDE is often tied to the platform on which the application is being developed. For example if building a .NET based application one is likely to use Visual Studio. If developing a J2EE application one may choose to use the eclipse as the IDE.

Having made a considerable investment of time and money in coming up with a process model – whether using COTS methodology such as RUP or your own in-house methodology – you would like all projects, irrespective of the IDE they use, to follow this process. This also includes getting access to the lessons learned across all projects enacting this process model.

All of this makes the already tough task of enacting software development processes in this mixed-technology development environment even more difficult. Rather than trying to convince developers to leave (step out to another interface) or at worse abandon (in favor of another IDE) their IDE simply to follow a process, it would make more sense to build process awareness into the tool they decide to use. The lack of a low-overhead process enactment interface should not become an excuse in not enacting the chosen process model.

The IRIS Plugin for Eclipse injects this process awareness into the eclipse IDE. Developers can now access the relevant process content without needing to leave the eclipse environment to dig through cumbersome process documentation. More importantly this information is available in a just-in-time manner when the project work items assigned to the developer are ready to be started. This means that there is zero additional overhead on part of the developer to lookup the process guidance associated with a project work item they are executing. In fact, if the process author has chosen to build a low-fidelity model prescribing a workflow based on activity state or work product state change events, the developers even get notifications when their work items are ready to be started. Developers see all project work items (with status) assigned to them and are able to distinguish pending work items. If needed they can create related work items as they progress in executing the work items assigned to them. When finished they can assign the work items to the next developer or other practioners that are part of the project being enacted using the defined process model.

IRIS Process Live acts as a unifying enactment platform across the enterprise. Here is an example of how with IRIS Process Live at the backend, the same process model can be used by developers and other practioners in disparate environments. In Figure 1a and 1b, the eclipse environment to view all project work items as well view a process links tab for one of the work items is shown. In Figure 2a and 2b, the same information is provided in a Visual Studio environment. Finally in Figure 3a and 3b, the same information is presented in a web view that may be used by practioners who don’t have desktop IDE clients on their machines.

Figure 1a - Eclipse

Figure 1b - Eclipse

Figure 2a - Visual Studio

Figure 2b - Visual Studio

Figure 3a - Web view

Figure 3b - Web view

The investments made in commercial methodologies such as RUP are realized by providing the rich method content to its intended audience. This low-impedance way to access process guidance - such as steps, guidance documents such as templates, best practices and patterns - without leaving the IDE environment increases the chances of project success.
The Osellus team is ready to share the work it has done in this area at the upcoming rational conference in Orlando. Come see us at booth 427 from 11-13 June 2007.

In my last post, I talked about the concepts of flexible process enactment as illustrated by the scientific research findings by John Noll et al. in this domain. Our first generation tooling for process enactment has already implemented these concepts that have been commercially offered for more than three years now. We have received numerous lessons learnt and other feedback from process mature organizations that have tried to implement this tooling in their development teams. Based on that we have come to a conclusion that flexible process enactment (as explained by John Noll et al.), although an essential first step needs to be further augmented for a successful process enactment in real-life projects.

There are two key aspects in our approach that incorporates the idea of a flexible instantiation of a process model. First, the model being followed is kept separate from the project work item instances which represent the instances of the process. That is, even though practioners have a view into the process model they are following, the project work items they are executing are handled independently by the tool. Second, practioners have a view into the model and the flexibility to create as many instances of whole/part of the process model as the project progresses.

Let me take an example to illustrate this approach. The left hand side of Figure 1 shows a work breakdown structure that has been created in a process modeling tool. Our first generation enactment tooling would have created a single instantiation of this process model resulting in a 1:1 mapping of process model elements and project work items. This is similar to a straightforward object-oriented modeling approach of instantiating a class. Unfortunately this also results in an unrealistic single monolithic instantiation of the complete process model that is unsuitable in a dynamic project environment consisting of knowledge workers as practioners.

Figure 1

In our new approach, the process model would serve as a reference to create project work items on the fly. This would need to address both – known unknowns and unknown unknowns. For example to handle “known unknowns” such as instantiating iterations, a practioner would simply instantiate those process elements that are needed at a given point of time in the project. In our example only the “Determine Requirements” work definition has been instantiated as part of “Iteration 0” in the first series or thread. Later on, practioners have opened another series that has instantiated “Iteration 0” complexly. Next, they have chosen to instantiate “Iteration 1..n” as part of two series. At any time, practioners can double click the process work breakdown element on the left hand side to study published process guidance for that element. Another scenario could have been where a change control workflow is triggered each time a change request is entered into the system. Again this is a known unknown since the model cannot predict how many times the change request is raised but it can define a reference workflow that may be triggered in complete or in part depending on the nature of every change request raised. The series or threads would facilitate concurrency in the project environment as these threads could be executed simultaneously.

Figure 2

The “unknown unknowns” are handled by allowing practioners to create additional project work items that do not map to any process element (since that knowledge was not known at time of modeling). For example as shown in Figure 2, I can create a related activity that although not defined by the process model is needed as part of the project and follows the activity “demonstrate the architecture”. The enactment system would enable us to track and log how these instances are created. One use of this information would be to act as lessons learned to improve the process models for other projects that are being implemented in a similar environment as this project.

I have come across an interesting set of papers from John Noll et al. talking about their research in the area of process enactment. The paper on “Flexible Process Enactment Using Low-Fidelity Models” brings out some important points.

Examining the reasons behind the lack of success of conventional workflow execution approaches in “dynamic, knowledge intensive activities” Noll contends that “In part, this is due to the dynamic nature of such activities: actors in knowledge-intensive environments continually adapt their activities to reflect increasing understanding of the problem at hand, which results from performing the knowledge-intensive activities.”

Software development is an excellent example of this “dynamic, knowledge intensive” environment and I agree with the lack of success in trying to adopt the traditional workflow approaches in software development projects. In fact in my opinion this is why tools designed for portfolio management, project management or business process execution have not been successful in the software process execution domain.

Noll then introduces the concept of “low-fidelity process models”. “These models specify a nominal order of tasks, but leave actors to carry out their activities as their expertise and situation dictates” he continues. “A low-fidelity process model does not seek to capture every detail and nuance of a knowledge-intensive process; rather, it documents the major activities of a process and the primary sequence in which they are performed”.

Noll then illustrates the benefits of the low-fidelity process model:

“
Low-fidelity models are easy to specify, and can be generated rapidly.
A low-fidelity model still captures the essential facets of a process, especially the resources consumed and artifacts produced by a given set of activities.
Because they seek to represent only high-level detail, low-fidelity models are relatively stable; that is, they continue to be accurate descriptions of the high-level process, even as the details of process activities evolve in response to knowledge and experience gained with the problem.
”

Again, I am in full agreement with the concept and benefits of low-fidelity process models. With our practical experience in implementing SPEM-based process models for software development we have seen that processes that can now be characterized as low-fidelity have resulted in wide uptake by practioners. Often, teams have been able to construct these low-fidelity process models from scratch from internal home-grown methodologies or by blending their internal best practices with commercial methodologies such as MSF, RUP, Macroscope and so on.

Noll then defines as enactment “driven by events, which are classified as either process events or resource events…. A predicate in the process description specifies the state that the required resources must satisfy before the activity can proceed. Process events signal action initiation and completion, and are generated directly by actors as a consequence of performing tasks. Resource events reflect changes in the environment, such as creation, deletion, and modification of resources, and time events such as deadline or alarms….. The enactment mechanism enables flexible enactment through the way it handles process and resource events.”

Again, I am in complete agreement with this view of enactment. Unfortunately SPEM (SPEM 2.0 is currently under finalization) does not go as far as we had hoped in supporting this concept. Without this support for concepts of states and state machine, process models built using SPEM 2.0 will remain un-enactable. This also implies that methodologies such as RUP and OpenUP that use the underlying SPEM meta-model would be un-enactable.

In my experience adding support for enactment is not an un-surmountable task. Our first generation enactment system, IRIS Process Live v1.0, already fully supported the above by providing activity and work product state-machines as well as pre-condition and goal. IRIS Process Live detects and triggers task and or work product state changes and sends notifications based on the work-flow relationships, assisting in implementing the process.