What is a process architecture ? My last post talked about a project manager tailoring a process or flexibly enacting a tailored process in a project. In this post I would like to zoom out and present how a typical process architecture looks like. Later I would share the litmus test that shows if your process architecture has been created correctly.

In a typical enterprise organization, the top layer in a process architecture consists of process elements. These are elements like roles, work products, activities that go on to constitute a process. These elements could belong to one or more commercial frameworks or could belong to different methodologies or frameworks. These could also include constitutional elements which the organization needs to define and declare in order to meet a internal or external control objective as part of regulatory or quality compliance.

The next layer would consist of process components that are fragments or clusters of related activities. These process components make it convenient to concentrate on specific aspects of a process. Depending on the business or compliancy objectives of the organization you could craft these components to achieve a certain result or describe a desired workflow. One or more components could be linked to create a larger component and eventually an end-to-end process.

Below the process element and process component layer you will have the process layer. This is where you will have the various sub-organizations or departments start crafting their processes in tandem with the software process engineering group. You would not only use elements from more than one packages but in all probability you will be working with other process authors in different time zones or geographical locations. This process layer also ensures that rather than relying on one monolithic process pushed by a vendor or consultant, teams can create processes drawing from different methodologies and process frameworks.

The next layer would make it convenient for the project management office to tailor the process for a category of projects. This domain layer makes it convenient for you to tailor processes for a technology or business domain. Some common examples I have seen in this layer are tailored processes for web applications or those for mobile applications. This domain specific tailoring makes it convenient for project managers to easily search and use processes in real projects. This takes us to the last and final layer, the engagement layer.

The engagement layer consists of processes that are tailored for a specific project. The project manager does not have to go any higher than the domain layer to look for a process that she can specialize for a project that is about to be initiated. Project managers already have their plate full with all the project start-up tasks and they don’t want any overhead associated with searching for and tailoring a process for their project. They want a tactical intuitive way to tailor a process for a project. If they end up spending more time than needed to prepare a process it is likely that this would end up costing more than the benefits accrued. So the litmus test for a successful process architecture is the time taken by a project manager to tailor and operationalize a process in a real project. The lower this transaction cost of tailoring and moving a process to enactment the higher marks you get for a well managed process architecture.

Almost no discussion on managing processes is complete without the use of the word “tailoring”. It is one of those words that although is understood commonly at a generic level starts giving trouble when applied without enough explanation in process development scenarios. I will attempt to describe how I understand “tailoring” in context of process models. I will provide this explanation using a practical scenario of applying a process in real projects.

A process is of little use unless it is expected to add value resulting in successful delivery of real life projects. Every endeavour or project, be it a software project, science experiment or manufacturing an article of clothing, has a certain process it follows. For us in the knowledge intensive domain of software development, successful project delivery is made tougher due to changing or late arising requirements coupled with involvement of skilled knowledge workers. My earlier post talks a bit more on these challenges. Over time, an organization would accumulate a large number of lessons learned as well as patterns and practices that work consistently well across projects. They will also have a collection of anti-processes (think anti-patterns) which they know don’t work. Some organizations would use a tool such as IRIS Process Author to structure their body of process knowledge for easy reuse.

Let us now add a temporal element relating to a point of time when an expected endeavour/project becomes known. At this time, you would study the characteristics of the known execution environment and try to modify your existing body of process knowledge to come up with a process model that best suits this expected project environment. As much as you can, you would also account for the known unknowns in your process model. Based on different situations, this would mean defining things like - a set of activities and their work breakdown structure or workflow, a set of work products that are created and or consumed in the process, the process roles that are staffed by practioners, the pre-post conditions that might be true within the process and so on. You may also defined the narrative guidance that would be read by the practioners in the project. This tailoring of the available process body of knowledge will result in bespoke processes suitable for a specific project execution environment. This process model would best describe the process framework that is expected to result in a successful execution of the project.

Now we will add the second temporal element relating to a point of time when the project is initiated. At this time you would use the tailored process to create instances of process definition elements as project work items. For example this would mean a project manager creating instances of a change control workflow as and when a change work item is added. I would call this instantiation of process definition as flexible process enactment. Keep in mind that the project manager can also add additional work items that are not defined in the process model. However, these changes to the process model could not have been anticipated as the time of process tailoring before the project is initiated. Some would argue that even this flexible process enactment is a “tailoring” of sorts since the process is being instantiated based on the unfolding project (and in some cases new work items are added to the process model). Personally, I have no problems with that as long as there is clarity on the context – tailoring before project initiation and tailoring after project initiation - is understood.

The key point I want to underline is that you would not leave process tailoring to be done after the project has started. Rather you would attempt to tailor your process, using the lessons learned from the past and the known unknowns of the project, before the project is initiated. Once the project starts you would flexibly instantiate your process elements as project work items driven by the circumstances of the unfolding project. Off course you will also add the unknown unknowns to your process. However these by their very definition would not have been known before project startup so could not have been modeled ahead of time.

In my experience process authoring for software development processes is a highly collaborative activity. It is unnatural to expect a solo effort to come up with a process that could be successfully consumed by a wide team of practioners. The process infrastructure also needs to scale well. Reusing existing process components not only reduces significant overhead but also makes the task of coming up with customized processes agile. Any tool that attempts to support process modeling needs to address these two objectives. Based on the evidence I have noticed, the EPF Composer (and RMC from IBM) fails in this regard.

From the information available, it appears to me that there are significant problems being faced by the Open UP team as they try to come up with a process model. The fundamental issue stems from the fact the EPF Composer is designed as a single-user client tool over-relying on SCM to check-in/check-out complicated process plugins. In my opinion it is unrealistic to manage processes in this manner. You basically end up choosing one of two undesirable options. The first option is to break up a process into multiple constituent plugins that are managed separately in an SCM. The second option is to put everything under one library that is checked in. Here is an on-going EPF-dev discussion thread on this topic. This is a thread questioning the rationale for having a overblown library with every sort of plugin in it.

Most enterprise teams will probably face similar problems when using Rational Method Composer (RMC). If the OpenUP team is having difficulties dealing with the limitations of the tool when trying to add a single (DSDM) plugin to the base OpenUP methodology, imagine the magnitude of problem in an enterprise environment when a blended methodology process architecture consists of a number of methods, regulatory and compliancy standards. Moreover, in that environment there will be a large number of subject matter experts who would likely want to collaborate as they contribute to various aspects of the processes. With the difficulties I am seeing with the usage of EPF Composer within the EPF team one can imagine the problems being intensified with scale.

Providing a server side support for process management is critical for a scalable process architecture and collaborative development of processes. This was a fundamental business objective for the Osellus engineering team that came up with a robust database-backed enterprise architecture for IRIS Process Author. Since every process asset resides in a database, it is easy for IRIS Process Author users to discover, tailor and export processes as needed. All this is done without resorting to complicated check-in/check-out rules associated with handling processes managed as files.

Finally, some would say that it is ironic that the very team that is promoting the “eclipse way” is actually building products that seemingly takes one down the path of single client development rather than team-based development or worse still providing only limited visibility into source process content.

In my experience I have seldom come across large enterprise organizations that mandate the use of a specific IDE while putting a blanket ban on any other IDE. Even within the same sub-organization, different development teams may have own their favorite IDEs. Moreover, the decision to use a particular IDE is often tied to the platform on which the application is being developed. For example if building a .NET based application one is likely to use Visual Studio. If developing a J2EE application one may choose to use the eclipse as the IDE.

Having made a considerable investment of time and money in coming up with a process model – whether using COTS methodology such as RUP or your own in-house methodology – you would like all projects, irrespective of the IDE they use, to follow this process. This also includes getting access to the lessons learned across all projects enacting this process model.

All of this makes the already tough task of enacting software development processes in this mixed-technology development environment even more difficult. Rather than trying to convince developers to leave (step out to another interface) or at worse abandon (in favor of another IDE) their IDE simply to follow a process, it would make more sense to build process awareness into the tool they decide to use. The lack of a low-overhead process enactment interface should not become an excuse in not enacting the chosen process model.

The IRIS Plugin for Eclipse injects this process awareness into the eclipse IDE. Developers can now access the relevant process content without needing to leave the eclipse environment to dig through cumbersome process documentation. More importantly this information is available in a just-in-time manner when the project work items assigned to the developer are ready to be started. This means that there is zero additional overhead on part of the developer to lookup the process guidance associated with a project work item they are executing. In fact, if the process author has chosen to build a low-fidelity model prescribing a workflow based on activity state or work product state change events, the developers even get notifications when their work items are ready to be started. Developers see all project work items (with status) assigned to them and are able to distinguish pending work items. If needed they can create related work items as they progress in executing the work items assigned to them. When finished they can assign the work items to the next developer or other practioners that are part of the project being enacted using the defined process model.

IRIS Process Live acts as a unifying enactment platform across the enterprise. Here is an example of how with IRIS Process Live at the backend, the same process model can be used by developers and other practioners in disparate environments. In Figure 1a and 1b, the eclipse environment to view all project work items as well view a process links tab for one of the work items is shown. In Figure 2a and 2b, the same information is provided in a Visual Studio environment. Finally in Figure 3a and 3b, the same information is presented in a web view that may be used by practioners who don’t have desktop IDE clients on their machines.

Figure 1a - Eclipse

Figure 1b - Eclipse

Figure 2a - Visual Studio

Figure 2b - Visual Studio

Figure 3a - Web view

Figure 3b - Web view

The investments made in commercial methodologies such as RUP are realized by providing the rich method content to its intended audience. This low-impedance way to access process guidance - such as steps, guidance documents such as templates, best practices and patterns - without leaving the IDE environment increases the chances of project success.
The Osellus team is ready to share the work it has done in this area at the upcoming rational conference in Orlando. Come see us at booth 427 from 11-13 June 2007.

In my last post, I talked about the concepts of flexible process enactment as illustrated by the scientific research findings by John Noll et al. in this domain. Our first generation tooling for process enactment has already implemented these concepts that have been commercially offered for more than three years now. We have received numerous lessons learnt and other feedback from process mature organizations that have tried to implement this tooling in their development teams. Based on that we have come to a conclusion that flexible process enactment (as explained by John Noll et al.), although an essential first step needs to be further augmented for a successful process enactment in real-life projects.

There are two key aspects in our approach that incorporates the idea of a flexible instantiation of a process model. First, the model being followed is kept separate from the project work item instances which represent the instances of the process. That is, even though practioners have a view into the process model they are following, the project work items they are executing are handled independently by the tool. Second, practioners have a view into the model and the flexibility to create as many instances of whole/part of the process model as the project progresses.

Let me take an example to illustrate this approach. The left hand side of Figure 1 shows a work breakdown structure that has been created in a process modeling tool. Our first generation enactment tooling would have created a single instantiation of this process model resulting in a 1:1 mapping of process model elements and project work items. This is similar to a straightforward object-oriented modeling approach of instantiating a class. Unfortunately this also results in an unrealistic single monolithic instantiation of the complete process model that is unsuitable in a dynamic project environment consisting of knowledge workers as practioners.

Figure 1

In our new approach, the process model would serve as a reference to create project work items on the fly. This would need to address both – known unknowns and unknown unknowns. For example to handle “known unknowns” such as instantiating iterations, a practioner would simply instantiate those process elements that are needed at a given point of time in the project. In our example only the “Determine Requirements” work definition has been instantiated as part of “Iteration 0” in the first series or thread. Later on, practioners have opened another series that has instantiated “Iteration 0” complexly. Next, they have chosen to instantiate “Iteration 1..n” as part of two series. At any time, practioners can double click the process work breakdown element on the left hand side to study published process guidance for that element. Another scenario could have been where a change control workflow is triggered each time a change request is entered into the system. Again this is a known unknown since the model cannot predict how many times the change request is raised but it can define a reference workflow that may be triggered in complete or in part depending on the nature of every change request raised. The series or threads would facilitate concurrency in the project environment as these threads could be executed simultaneously.

Figure 2

The “unknown unknowns” are handled by allowing practioners to create additional project work items that do not map to any process element (since that knowledge was not known at time of modeling). For example as shown in Figure 2, I can create a related activity that although not defined by the process model is needed as part of the project and follows the activity “demonstrate the architecture”. The enactment system would enable us to track and log how these instances are created. One use of this information would be to act as lessons learned to improve the process models for other projects that are being implemented in a similar environment as this project.

I have come across an interesting set of papers from John Noll et al. talking about their research in the area of process enactment. The paper on “Flexible Process Enactment Using Low-Fidelity Models” brings out some important points.

Examining the reasons behind the lack of success of conventional workflow execution approaches in “dynamic, knowledge intensive activities” Noll contends that “In part, this is due to the dynamic nature of such activities: actors in knowledge-intensive environments continually adapt their activities to reflect increasing understanding of the problem at hand, which results from performing the knowledge-intensive activities.”

Software development is an excellent example of this “dynamic, knowledge intensive” environment and I agree with the lack of success in trying to adopt the traditional workflow approaches in software development projects. In fact in my opinion this is why tools designed for portfolio management, project management or business process execution have not been successful in the software process execution domain.

Noll then introduces the concept of “low-fidelity process models”. “These models specify a nominal order of tasks, but leave actors to carry out their activities as their expertise and situation dictates” he continues. “A low-fidelity process model does not seek to capture every detail and nuance of a knowledge-intensive process; rather, it documents the major activities of a process and the primary sequence in which they are performed”.

Noll then illustrates the benefits of the low-fidelity process model:

“
Low-fidelity models are easy to specify, and can be generated rapidly.
A low-fidelity model still captures the essential facets of a process, especially the resources consumed and artifacts produced by a given set of activities.
Because they seek to represent only high-level detail, low-fidelity models are relatively stable; that is, they continue to be accurate descriptions of the high-level process, even as the details of process activities evolve in response to knowledge and experience gained with the problem.
”

Again, I am in full agreement with the concept and benefits of low-fidelity process models. With our practical experience in implementing SPEM-based process models for software development we have seen that processes that can now be characterized as low-fidelity have resulted in wide uptake by practioners. Often, teams have been able to construct these low-fidelity process models from scratch from internal home-grown methodologies or by blending their internal best practices with commercial methodologies such as MSF, RUP, Macroscope and so on.

Noll then defines as enactment “driven by events, which are classified as either process events or resource events…. A predicate in the process description specifies the state that the required resources must satisfy before the activity can proceed. Process events signal action initiation and completion, and are generated directly by actors as a consequence of performing tasks. Resource events reflect changes in the environment, such as creation, deletion, and modification of resources, and time events such as deadline or alarms….. The enactment mechanism enables flexible enactment through the way it handles process and resource events.”

Again, I am in complete agreement with this view of enactment. Unfortunately SPEM (SPEM 2.0 is currently under finalization) does not go as far as we had hoped in supporting this concept. Without this support for concepts of states and state machine, process models built using SPEM 2.0 will remain un-enactable. This also implies that methodologies such as RUP and OpenUP that use the underlying SPEM meta-model would be un-enactable.

In my experience adding support for enactment is not an un-surmountable task. Our first generation enactment system, IRIS Process Live v1.0, already fully supported the above by providing activity and work product state-machines as well as pre-condition and goal. IRIS Process Live detects and triggers task and or work product state changes and sends notifications based on the work-flow relationships, assisting in implementing the process.

IBM is preparing to release RPM v 7.1 next month:

“IBM Rational Portfolio Manager v.7.1 is an enterprise project management solution that can be used by everyone in an organization - CIO, project managers, business analysts, developers - to balance IT investments and identify service priorities. A new Web 2.0 interface allows team members to manage their work, submit time and expense reports, and a tight integration between Rational Portfolio Manager and IBM’s Rational portfolio helps ensure software projects are conceived, built, tested, deployed and provisioned on schedule. An analytical component to Rational Portfolio Manager enables automated decision and approval cycles for software development teams while finding trends and overlaps in projects. Project teams can increase efficiency with new templates and best practices that generate compliance reports and ease project start up.”

Source: http://www.ebizq.net/news/8035.html

From what I read, IBM is trying to make this RPM dot release appear as a process enactment solution for developers while keeping the focus on solving a compliancy need. In my view this is a wrong approach to take and does not make sense at all. In fact this is completely opposite to the second generation process enactment tooling approach Osellus is taking. Why do I say this?

As I have mentioned in my earlier blog posts, a command and control approach to software process has proven to be a failure. Our approach is to reduce the overhead associated to process enactment. Process should be instrumented into the developer tools (IDE) delivered in a non-intrusive friction-free manner. Our second generation tooling on process enactment is bringing us closer to the ultimate objective of making process invisible during a project. Instead, from the information I have seen till now, IBM has chosen to take a mature portfolio management tool and as part of a dot release tried to inject process enactment into it. Developers who had no reason to use this tool will now be asked to update it during project enactment. I don’t see this working. In fact just yesterday I was in a meeting with one of the largest banks in Canada where they mentioned that their current investment in a portfolio management tool is underutilized. Project managers and other management folks do not want to go through the hassle of managing yet another tool simply to follow a process and get compliancy reports.

Osellus will continue to work with its partners to ensure that we take the practioners tools (such as IDE) and make them process-aware rather than imposing another tool simply to follow a process and compliancy reporting.

Process mature organizations define and tailor processes at various organizational unit or domain levels. IRIS Process Author (IPA) provides the tooling to manage process assets structured within the enterprise process architecture. Apart from subject matter experts and process engineers, many project managers tailor processes based on project execution environment - for a specific engagement or a specific project. A process can be tailored by a process engineer or a project manager to better suit a particular project or category of projects. In addition to process tailoring there is a need for a flexible, on-demand instantiation of tailored process anytime during enactment. This is because not all aspects of the process usage are known ahead of time. For example, number of different iterations only becomes apparent as the project progresses. Also this gives the flexibility of re-instantiating any part of the process as needed. For example, the workflow for change request can be instantiated any number of times, anytime during the project.

Process tailoring only takes place before start of the project. Flexible, on-demand instantiation of process only happens after the start of project.

Process element instantiation is the creation of executable project work items that drive process enactment in real projects. In first generation enactment systems such as IRIS Process Live v1.0, there was a 1:1 mapping between process definition elements and process enactment elements. Based on experiences and lessons learned this has proven to be a rigid and “artificial” way of looking at process enactment. Our second generation enactment tooling addresses this project team’s need to create project work items in an “as-many” and “any-where” manner.

Project work items may be created in response to either/both of the following two conditions. In the first case, the project team simply creates as many instances of project work items as needed by the progress they have made at any time during the project. In this case the process definition elements are instantiated to create these project work items. For example teams decide on the number of iterations needed in the midst of a live project, IRIS Process Live would let them decide on the composition and quantity of iterations in a flexible adhoc manner. In this case even though teams have full flexibility in creating these instances, they are still following the prescribed process since they create instances of process elements from the pre-defined process definition

Although it is expected that the teams follow the defined process to create these instances, it is not uncommon that they may create additional process enactment elements, for example, adding a “review” activity after use cases were refined that was missing in the process definition. In this second case, the project team has the ability to add missing process definition elements (and treating them as process enactment elements) without giving up on the original process definition completely.

In both cases, the process element instantiation is tracked to prepare reports on the type and nature of instantiation during a project lifecycle a well as overall process compliance. To facilitate process improvement, lessons learned by tracking the number and nature of project work item instances creation is also studied by process engineers.

I will continue to share similar advancement in the state of art around process enactment and how our tooling continues to evolve using the feedback from an expanding community of process mature enterprise organizations.

Process mature organizations do not think process is a dirty word. Unfortunately many other organizations have been victims of half-hearted attempts to simply adopt the flavour-of-the-day methodology as their de-facto process without adequate focus on enactment.

My experience working with process mature enterprise organizations has been that a successful process definition really starts when you start thinking about the enactment scenarios it is trying to address. Simply put, convincing a business sponsor that processes are important and worth investing into, is easier when you articulate the advantages it will have on project success. Moreover this assessment has to be data-based and not subjective. Let me take an example from software development to further detail this idea.

It is now widely accepted that design and development go hand in hand. After a designer comes up with software design it is not uncommon (in fact it is expected) for the development team to uncover gaps and further improve design as they go about implementing it. It would be unrealistic to expect the design document to be complete even before writing the first line of code based on it. The same applies to the world of process definition and enactment.

In my view it is not possible to come up with a sustainable process definition without enacting it in real projects. Unless project teams try to enact the defined processes, they cannot uncover process gaps. Unfortunately this is also the reason why in many organizations process has become a dirty word. There have been many anecdotal stories about how someone (usually a consultant from or trained in the latest methodology fad) would create a process (mostly solo with just token efforts to involve all stakeholders) and toss that over to the development team, collect his/her cheque and leave. At best the knowledge workers (I would characterize all practioners in a software development project as knowledge workers) would take a look at this process binder and throw it over to the trash bin. At worst they would try to follow the process in the absence of a process enactment framework in place, fail, and become demotivated vowing never to follow similar imposed processes in future. In both cases the project schedule languishes and help of “hero” practioners is needed to save the project. But how are process mature companies managing their processes and enactment?

To begin with, these enterprises realize that process creation should not be a single user effort. The process management infrastructure should lower the barrier of participation by enabling all stakeholders to collaboratively come up with processes. For an enterprise with a number of teams spread across different time zones this would mean:
- Collaborating while managing a process infrastructure built on a commonly understood, uncomplicated process meta-model that supports all methodologies including home grown.
- Supporting the creation of a process architecture that would map to the organization structure with access control granting visibility into process assets and promote reuse.
- Representing process assets in relational databases allowing subject matter experts to keep updating the vast pool of best practices that is impossible to manage in a file-based system.
- Provisioning a visual, intuitive user interface to create processes. No one wants to spend weeks to understand how to extend an activity.

Putting in place a process management infrastructure is just half the story. Process mature organizations would ensure that the processes created in a manner defined above are validated for enactment. This means an automated mechanism to check for holes in the process, such as usage of work products that become outdated as the process was being developed. Having done that it is time to enact these processes. It is during process enactment that teams would uncover what is missing in their processes. Lessons learned as part of a successful or unsuccessful implementation would provide rich data to improve these processes and institutionalize the lessons across the enterprise. Compliancy reporting would mean teams can self-audit how far/away are they from meeting the compliance points as laid out in the defined process. Finally, with data-based process interventions and the empowerment of project practioners the chances of project success keep getting higher with successive implementations.

So why has Osellus invested more than four years in this process enactment domain and is introducing our second generation enactment tooling later this month. Here are a few reasons:

- Process enactment is emerging as the new frontier in software development process world. Standards such as ISO/IEC 24744 already taking lead in this area. The Osellus SPEM 2.0 submission had enactment as a cornerstone of the process meta-model. There is a strong demand from process mature organizations to address this need.
- Enables wider (more practitioners empowered with buy-in) and deeper (actual “consumption” of processes in real projects) process adoption.
- Increases chances of better software quality since the best practices and guidances are viewed in terms of enactment/endeavor domain on real projects and not just published binders/websites.
- Lays the groundwork for gathering metrics on project health, process compliancy (regulatory and quality frameworks) and data-based process improvement decisions.

In a later post, I am going to share a bit more about the second generation process enactment infrastructure we have built. This draws from direct feedback from enterprise teams as well as our own lessons learned from our first generation tooling around enactment.

Let’s face it. Many of us have worked in teams/organizations that at some time or the other have said “lets-fix-the-processes-we-follow-around-here”. Unfortunately, based on my knowledge most of these initiatives don’t live up to their promise (or the money spent on them!) despite the investments in consultants, methodology-licenses or books. It is not that we don’t have the skills to come up with a process. It is the nature of software development which in most enterprises is a collaborative team based activity that has proven to be difficult to predict (I wont go into all the reasons here - changing requirements, schedules, technology, nature of knowledge workers). A static one-time (often one-person!) effort to come up with a process and then throwing it over to the wider team (in the form of documents or published websites) is doomed from the get-go. In fact in my personal view, in the absence of sustainable plan to collaboratively manage and then operationalize your process assets, this is a colossal waste of resources.

Over the next few weeks and months I will outline how process-mature organizations can take a pioneering role by setting up a sustainable process infrastructure that results in demonstrable gains to their project sponsors/stakeholders. I will also list some organizations that are in the process of doing so or are simply taking a tactical decision to recoup their costly investments in methodology by moving to operationalize them.