In my opinion Jazz is the star of this years rational conference. Take a look at some of the pictures from the conference and the keynote introducing jazz. Erich Gamma did an excellent presentation on jazz during the keynote and later during a dedicated technology session on Jazz.

Me next to the Jazz Live poster

The Jazz show begins…

(From left to right) Chandra, Payman and Dave

(From left to right) Dave, Payman, Me, Chandra

There were many exciting things I saw that vindicated some of our fundamental beliefs in the software process automation domain. First with Microsoft’s VSTS and now with IBM Jazz there is overwhelming evidence that low-fidelity software development process models are the cornerstone of collaborative friction-free process enactment. This enactment is not only helpful (just-in-time, just-enough) for developers and other project practioners but is also essential for compliance requirements as well as ultimately, feedback to improve processes. The end goal is “continuous”, successful project delivery. Although there are still a lot of unanswered questions on the Jazz strategy, one thing that is clear is that it is a fundamental shift in the way IBM rational has been building and deploying its developer tools.

We are getting ready to participate at the IBM rational software development conference 2007 in Orlando from June 10-14th, 2007.

If you are at the conference, please do stop by our booth (#427) to see some of the second generation tooling we are going to be demonstrating.

We will reach Orlando Monday afternoon (11th June) and I will be blogging live from the conference on all the exciting action.

In my experience I have seldom come across large enterprise organizations that mandate the use of a specific IDE while putting a blanket ban on any other IDE. Even within the same sub-organization, different development teams may have own their favorite IDEs. Moreover, the decision to use a particular IDE is often tied to the platform on which the application is being developed. For example if building a .NET based application one is likely to use Visual Studio. If developing a J2EE application one may choose to use the eclipse as the IDE.

Having made a considerable investment of time and money in coming up with a process model – whether using COTS methodology such as RUP or your own in-house methodology – you would like all projects, irrespective of the IDE they use, to follow this process. This also includes getting access to the lessons learned across all projects enacting this process model.

All of this makes the already tough task of enacting software development processes in this mixed-technology development environment even more difficult. Rather than trying to convince developers to leave (step out to another interface) or at worse abandon (in favor of another IDE) their IDE simply to follow a process, it would make more sense to build process awareness into the tool they decide to use. The lack of a low-overhead process enactment interface should not become an excuse in not enacting the chosen process model.

The IRIS Plugin for Eclipse injects this process awareness into the eclipse IDE. Developers can now access the relevant process content without needing to leave the eclipse environment to dig through cumbersome process documentation. More importantly this information is available in a just-in-time manner when the project work items assigned to the developer are ready to be started. This means that there is zero additional overhead on part of the developer to lookup the process guidance associated with a project work item they are executing. In fact, if the process author has chosen to build a low-fidelity model prescribing a workflow based on activity state or work product state change events, the developers even get notifications when their work items are ready to be started. Developers see all project work items (with status) assigned to them and are able to distinguish pending work items. If needed they can create related work items as they progress in executing the work items assigned to them. When finished they can assign the work items to the next developer or other practioners that are part of the project being enacted using the defined process model.

IRIS Process Live acts as a unifying enactment platform across the enterprise. Here is an example of how with IRIS Process Live at the backend, the same process model can be used by developers and other practioners in disparate environments. In Figure 1a and 1b, the eclipse environment to view all project work items as well view a process links tab for one of the work items is shown. In Figure 2a and 2b, the same information is provided in a Visual Studio environment. Finally in Figure 3a and 3b, the same information is presented in a web view that may be used by practioners who don’t have desktop IDE clients on their machines.

Figure 1a - Eclipse

Figure 1b - Eclipse

Figure 2a - Visual Studio

Figure 2b - Visual Studio

Figure 3a - Web view

Figure 3b - Web view

The investments made in commercial methodologies such as RUP are realized by providing the rich method content to its intended audience. This low-impedance way to access process guidance - such as steps, guidance documents such as templates, best practices and patterns - without leaving the IDE environment increases the chances of project success.
The Osellus team is ready to share the work it has done in this area at the upcoming rational conference in Orlando. Come see us at booth 427 from 11-13 June 2007.

In my last post, I talked about the concepts of flexible process enactment as illustrated by the scientific research findings by John Noll et al. in this domain. Our first generation tooling for process enactment has already implemented these concepts that have been commercially offered for more than three years now. We have received numerous lessons learnt and other feedback from process mature organizations that have tried to implement this tooling in their development teams. Based on that we have come to a conclusion that flexible process enactment (as explained by John Noll et al.), although an essential first step needs to be further augmented for a successful process enactment in real-life projects.

There are two key aspects in our approach that incorporates the idea of a flexible instantiation of a process model. First, the model being followed is kept separate from the project work item instances which represent the instances of the process. That is, even though practioners have a view into the process model they are following, the project work items they are executing are handled independently by the tool. Second, practioners have a view into the model and the flexibility to create as many instances of whole/part of the process model as the project progresses.

Let me take an example to illustrate this approach. The left hand side of Figure 1 shows a work breakdown structure that has been created in a process modeling tool. Our first generation enactment tooling would have created a single instantiation of this process model resulting in a 1:1 mapping of process model elements and project work items. This is similar to a straightforward object-oriented modeling approach of instantiating a class. Unfortunately this also results in an unrealistic single monolithic instantiation of the complete process model that is unsuitable in a dynamic project environment consisting of knowledge workers as practioners.

Figure 1

In our new approach, the process model would serve as a reference to create project work items on the fly. This would need to address both – known unknowns and unknown unknowns. For example to handle “known unknowns” such as instantiating iterations, a practioner would simply instantiate those process elements that are needed at a given point of time in the project. In our example only the “Determine Requirements” work definition has been instantiated as part of “Iteration 0” in the first series or thread. Later on, practioners have opened another series that has instantiated “Iteration 0” complexly. Next, they have chosen to instantiate “Iteration 1..n” as part of two series. At any time, practioners can double click the process work breakdown element on the left hand side to study published process guidance for that element. Another scenario could have been where a change control workflow is triggered each time a change request is entered into the system. Again this is a known unknown since the model cannot predict how many times the change request is raised but it can define a reference workflow that may be triggered in complete or in part depending on the nature of every change request raised. The series or threads would facilitate concurrency in the project environment as these threads could be executed simultaneously.

Figure 2

The “unknown unknowns” are handled by allowing practioners to create additional project work items that do not map to any process element (since that knowledge was not known at time of modeling). For example as shown in Figure 2, I can create a related activity that although not defined by the process model is needed as part of the project and follows the activity “demonstrate the architecture”. The enactment system would enable us to track and log how these instances are created. One use of this information would be to act as lessons learned to improve the process models for other projects that are being implemented in a similar environment as this project.

I have come across an interesting set of papers from John Noll et al. talking about their research in the area of process enactment. The paper on “Flexible Process Enactment Using Low-Fidelity Models” brings out some important points.

Examining the reasons behind the lack of success of conventional workflow execution approaches in “dynamic, knowledge intensive activities” Noll contends that “In part, this is due to the dynamic nature of such activities: actors in knowledge-intensive environments continually adapt their activities to reflect increasing understanding of the problem at hand, which results from performing the knowledge-intensive activities.”

Software development is an excellent example of this “dynamic, knowledge intensive” environment and I agree with the lack of success in trying to adopt the traditional workflow approaches in software development projects. In fact in my opinion this is why tools designed for portfolio management, project management or business process execution have not been successful in the software process execution domain.

Noll then introduces the concept of “low-fidelity process models”. “These models specify a nominal order of tasks, but leave actors to carry out their activities as their expertise and situation dictates” he continues. “A low-fidelity process model does not seek to capture every detail and nuance of a knowledge-intensive process; rather, it documents the major activities of a process and the primary sequence in which they are performed”.

Noll then illustrates the benefits of the low-fidelity process model:

“
Low-fidelity models are easy to specify, and can be generated rapidly.
A low-fidelity model still captures the essential facets of a process, especially the resources consumed and artifacts produced by a given set of activities.
Because they seek to represent only high-level detail, low-fidelity models are relatively stable; that is, they continue to be accurate descriptions of the high-level process, even as the details of process activities evolve in response to knowledge and experience gained with the problem.
”

Again, I am in full agreement with the concept and benefits of low-fidelity process models. With our practical experience in implementing SPEM-based process models for software development we have seen that processes that can now be characterized as low-fidelity have resulted in wide uptake by practioners. Often, teams have been able to construct these low-fidelity process models from scratch from internal home-grown methodologies or by blending their internal best practices with commercial methodologies such as MSF, RUP, Macroscope and so on.

Noll then defines as enactment “driven by events, which are classified as either process events or resource events…. A predicate in the process description specifies the state that the required resources must satisfy before the activity can proceed. Process events signal action initiation and completion, and are generated directly by actors as a consequence of performing tasks. Resource events reflect changes in the environment, such as creation, deletion, and modification of resources, and time events such as deadline or alarms….. The enactment mechanism enables flexible enactment through the way it handles process and resource events.”

Again, I am in complete agreement with this view of enactment. Unfortunately SPEM (SPEM 2.0 is currently under finalization) does not go as far as we had hoped in supporting this concept. Without this support for concepts of states and state machine, process models built using SPEM 2.0 will remain un-enactable. This also implies that methodologies such as RUP and OpenUP that use the underlying SPEM meta-model would be un-enactable.

In my experience adding support for enactment is not an un-surmountable task. Our first generation enactment system, IRIS Process Live v1.0, already fully supported the above by providing activity and work product state-machines as well as pre-condition and goal. IRIS Process Live detects and triggers task and or work product state changes and sends notifications based on the work-flow relationships, assisting in implementing the process.

IBM is preparing to release RPM v 7.1 next month:

“IBM Rational Portfolio Manager v.7.1 is an enterprise project management solution that can be used by everyone in an organization - CIO, project managers, business analysts, developers - to balance IT investments and identify service priorities. A new Web 2.0 interface allows team members to manage their work, submit time and expense reports, and a tight integration between Rational Portfolio Manager and IBM’s Rational portfolio helps ensure software projects are conceived, built, tested, deployed and provisioned on schedule. An analytical component to Rational Portfolio Manager enables automated decision and approval cycles for software development teams while finding trends and overlaps in projects. Project teams can increase efficiency with new templates and best practices that generate compliance reports and ease project start up.”

Source: http://www.ebizq.net/news/8035.html

From what I read, IBM is trying to make this RPM dot release appear as a process enactment solution for developers while keeping the focus on solving a compliancy need. In my view this is a wrong approach to take and does not make sense at all. In fact this is completely opposite to the second generation process enactment tooling approach Osellus is taking. Why do I say this?

As I have mentioned in my earlier blog posts, a command and control approach to software process has proven to be a failure. Our approach is to reduce the overhead associated to process enactment. Process should be instrumented into the developer tools (IDE) delivered in a non-intrusive friction-free manner. Our second generation tooling on process enactment is bringing us closer to the ultimate objective of making process invisible during a project. Instead, from the information I have seen till now, IBM has chosen to take a mature portfolio management tool and as part of a dot release tried to inject process enactment into it. Developers who had no reason to use this tool will now be asked to update it during project enactment. I don’t see this working. In fact just yesterday I was in a meeting with one of the largest banks in Canada where they mentioned that their current investment in a portfolio management tool is underutilized. Project managers and other management folks do not want to go through the hassle of managing yet another tool simply to follow a process and get compliancy reports.

Osellus will continue to work with its partners to ensure that we take the practioners tools (such as IDE) and make them process-aware rather than imposing another tool simply to follow a process and compliancy reporting.

Process mature organizations define and tailor processes at various organizational unit or domain levels. IRIS Process Author (IPA) provides the tooling to manage process assets structured within the enterprise process architecture. Apart from subject matter experts and process engineers, many project managers tailor processes based on project execution environment - for a specific engagement or a specific project. A process can be tailored by a process engineer or a project manager to better suit a particular project or category of projects. In addition to process tailoring there is a need for a flexible, on-demand instantiation of tailored process anytime during enactment. This is because not all aspects of the process usage are known ahead of time. For example, number of different iterations only becomes apparent as the project progresses. Also this gives the flexibility of re-instantiating any part of the process as needed. For example, the workflow for change request can be instantiated any number of times, anytime during the project.

Process tailoring only takes place before start of the project. Flexible, on-demand instantiation of process only happens after the start of project.

Process element instantiation is the creation of executable project work items that drive process enactment in real projects. In first generation enactment systems such as IRIS Process Live v1.0, there was a 1:1 mapping between process definition elements and process enactment elements. Based on experiences and lessons learned this has proven to be a rigid and “artificial” way of looking at process enactment. Our second generation enactment tooling addresses this project team’s need to create project work items in an “as-many” and “any-where” manner.

Project work items may be created in response to either/both of the following two conditions. In the first case, the project team simply creates as many instances of project work items as needed by the progress they have made at any time during the project. In this case the process definition elements are instantiated to create these project work items. For example teams decide on the number of iterations needed in the midst of a live project, IRIS Process Live would let them decide on the composition and quantity of iterations in a flexible adhoc manner. In this case even though teams have full flexibility in creating these instances, they are still following the prescribed process since they create instances of process elements from the pre-defined process definition

Although it is expected that the teams follow the defined process to create these instances, it is not uncommon that they may create additional process enactment elements, for example, adding a “review” activity after use cases were refined that was missing in the process definition. In this second case, the project team has the ability to add missing process definition elements (and treating them as process enactment elements) without giving up on the original process definition completely.

In both cases, the process element instantiation is tracked to prepare reports on the type and nature of instantiation during a project lifecycle a well as overall process compliance. To facilitate process improvement, lessons learned by tracking the number and nature of project work item instances creation is also studied by process engineers.

I will continue to share similar advancement in the state of art around process enactment and how our tooling continues to evolve using the feedback from an expanding community of process mature enterprise organizations.

Process mature organizations do not think process is a dirty word. Unfortunately many other organizations have been victims of half-hearted attempts to simply adopt the flavour-of-the-day methodology as their de-facto process without adequate focus on enactment.

My experience working with process mature enterprise organizations has been that a successful process definition really starts when you start thinking about the enactment scenarios it is trying to address. Simply put, convincing a business sponsor that processes are important and worth investing into, is easier when you articulate the advantages it will have on project success. Moreover this assessment has to be data-based and not subjective. Let me take an example from software development to further detail this idea.

It is now widely accepted that design and development go hand in hand. After a designer comes up with software design it is not uncommon (in fact it is expected) for the development team to uncover gaps and further improve design as they go about implementing it. It would be unrealistic to expect the design document to be complete even before writing the first line of code based on it. The same applies to the world of process definition and enactment.

In my view it is not possible to come up with a sustainable process definition without enacting it in real projects. Unless project teams try to enact the defined processes, they cannot uncover process gaps. Unfortunately this is also the reason why in many organizations process has become a dirty word. There have been many anecdotal stories about how someone (usually a consultant from or trained in the latest methodology fad) would create a process (mostly solo with just token efforts to involve all stakeholders) and toss that over to the development team, collect his/her cheque and leave. At best the knowledge workers (I would characterize all practioners in a software development project as knowledge workers) would take a look at this process binder and throw it over to the trash bin. At worst they would try to follow the process in the absence of a process enactment framework in place, fail, and become demotivated vowing never to follow similar imposed processes in future. In both cases the project schedule languishes and help of “hero” practioners is needed to save the project. But how are process mature companies managing their processes and enactment?

To begin with, these enterprises realize that process creation should not be a single user effort. The process management infrastructure should lower the barrier of participation by enabling all stakeholders to collaboratively come up with processes. For an enterprise with a number of teams spread across different time zones this would mean:
- Collaborating while managing a process infrastructure built on a commonly understood, uncomplicated process meta-model that supports all methodologies including home grown.
- Supporting the creation of a process architecture that would map to the organization structure with access control granting visibility into process assets and promote reuse.
- Representing process assets in relational databases allowing subject matter experts to keep updating the vast pool of best practices that is impossible to manage in a file-based system.
- Provisioning a visual, intuitive user interface to create processes. No one wants to spend weeks to understand how to extend an activity.

Putting in place a process management infrastructure is just half the story. Process mature organizations would ensure that the processes created in a manner defined above are validated for enactment. This means an automated mechanism to check for holes in the process, such as usage of work products that become outdated as the process was being developed. Having done that it is time to enact these processes. It is during process enactment that teams would uncover what is missing in their processes. Lessons learned as part of a successful or unsuccessful implementation would provide rich data to improve these processes and institutionalize the lessons across the enterprise. Compliancy reporting would mean teams can self-audit how far/away are they from meeting the compliance points as laid out in the defined process. Finally, with data-based process interventions and the empowerment of project practioners the chances of project success keep getting higher with successive implementations.

So why has Osellus invested more than four years in this process enactment domain and is introducing our second generation enactment tooling later this month. Here are a few reasons:

- Process enactment is emerging as the new frontier in software development process world. Standards such as ISO/IEC 24744 already taking lead in this area. The Osellus SPEM 2.0 submission had enactment as a cornerstone of the process meta-model. There is a strong demand from process mature organizations to address this need.
- Enables wider (more practitioners empowered with buy-in) and deeper (actual “consumption” of processes in real projects) process adoption.
- Increases chances of better software quality since the best practices and guidances are viewed in terms of enactment/endeavor domain on real projects and not just published binders/websites.
- Lays the groundwork for gathering metrics on project health, process compliancy (regulatory and quality frameworks) and data-based process improvement decisions.

In a later post, I am going to share a bit more about the second generation process enactment infrastructure we have built. This draws from direct feedback from enterprise teams as well as our own lessons learned from our first generation tooling around enactment.

Let’s face it. Many of us have worked in teams/organizations that at some time or the other have said “lets-fix-the-processes-we-follow-around-here”. Unfortunately, based on my knowledge most of these initiatives don’t live up to their promise (or the money spent on them!) despite the investments in consultants, methodology-licenses or books. It is not that we don’t have the skills to come up with a process. It is the nature of software development which in most enterprises is a collaborative team based activity that has proven to be difficult to predict (I wont go into all the reasons here - changing requirements, schedules, technology, nature of knowledge workers). A static one-time (often one-person!) effort to come up with a process and then throwing it over to the wider team (in the form of documents or published websites) is doomed from the get-go. In fact in my personal view, in the absence of sustainable plan to collaboratively manage and then operationalize your process assets, this is a colossal waste of resources.

Over the next few weeks and months I will outline how process-mature organizations can take a pioneering role by setting up a sustainable process infrastructure that results in demonstrable gains to their project sponsors/stakeholders. I will also list some organizations that are in the process of doing so or are simply taking a tactical decision to recoup their costly investments in methodology by moving to operationalize them.

Microsoft has just unveiled Silverlight.

“Silverlight is a cross-browser, cross-platform plug-in for delivering the next generation of media experiences and rich interactive applications (RIAs) for the Web.”

For product companies that rely on rich interactive applications in their tools/services this is a significent development and I am sure they are are going to closely follow the details being released by Microsoft over the next few weeks.