Last night I had a chance to attend a Jazz demonstration by John Wiegand at CASCON 07. It is always a treat to watch John speak and yesterday’s session was no different. For those who have not seen John and Erich speak about collaborative development, here is a short video to one of their previous videos talking about open source development.

John gave a glimpse of the current state of Jazz technology. Although Erich was unable to join the demonstration concurrently from OOPSLA in Montreal as planned, judging from the reaction of the standing room only audience the session was well received. There are several cool things that were shown including the concept of personal work spaces that amongst other things let developers see the effects of checkins before touching the main code repository. John ran through a scenario in which two new developers join an existing project. The system configured the tool behavior for these newcomers based on the process definition. Living up to the promise of transparency, these newcomer developers get access to all work items in their project and across different projects. They then get down to the brass tracks by working on their own work items while engaging in adhoc collaboration and sharing changes with others in the team.

I had a chance to chat with John after the demonstration. Since my focus was primarily on understanding the process awareness built into this integrated toolset, John was quick to point out their definition of process in Jazz. Process in Jazz means the policy and configuration settings that govern tool behavior. The process definition such as checkin polices and build management can be customized for individual projects. We then had a candid discussion on the need to have balance between having a process definition that spawns project work items without burdening developers with complicated process terminology. I can’t go onto all the details here but we agreed that over-complicated process models (based on over-engineered meta-models that require a steep learning curve) could be harmful to the concept of collaborative development environments. This is the reason the Jazz team wants to be very careful when they talk about process and their definition of it.

John was also generous to help me with a favour. He has promised to see how can he make the EclipseWay available to the open source community. This will mean anyone can take this collection of best practices and mash them up with their own in-house process description or third party process definitions without getting a knock on their door by a team of IBM lawyers. I know many people in the EPF community had been asking for such a thing and will join me in thanking John for helping us with this.

I am back at CASCON on Thursday to join a BOF session on Jazz.

In my view IBM is struggling to align their products and practices with the advancement in the state of around software development process automation and collaborative development environment.

In this post I am going to focus on their release of Rational Method Composer 7.2 (RMC 7.2) and specifically point out why it is difficult for me to be convinced that they have done a root cause analysis of their previous failures (with RPW or earlier versions of RMC) or have recognized the direction where software process automation is headed.

When one thinks of a product release from IBM you would obviously expect a high level of maturity, sophistication and thought leadership. IBM is universally well regarded and respected for pushing concepts like SOA and software-as-service. However, when you look at how the Rational team has implemented the tooling around creating and tailoring processes, one wonders why that thought leadership is not being put in practice here. By choosing to keep RMC 7.2 as a thick client application the rational team seems to be stuck in a time warp. I simply cannot imagine a scenario where an enterprise is told that its process assets (which in many cases have the “secret sauce” of project success) is encased in a few laptops. So I was bit surprised that even while acknowledging this problem of scalability and multi-methodology blending, the rational team has chosen to resolve it by continuing to stick to the flawed design decision of using SCM to check-in/check-out process plug-ins. But now, to make matters even more difficult, process authors (often non-programmers), are required to use the eclipse IDE environment to derive some scalability benefits. This is simply unbelievable. First of all, this still does not solve the problem of scalability (see why). Secondly, SCM is a code repository specially built for handling code and not process assets. Finally there is little logic in making process authoring so complicated that it requires a team of consultants to even setup the multi-layer environment for process authoring. Moreover in this day and age when even videos are being edited online asking a process team to work in this client-only disconnected mode is simply irresponsible. I did notice that the rational team did take note of the overwhelming negative feedback on usability of RMC and I welcome that. As in IBM partner and while exhibiting at the RSDC in June 07 we heard lot of this negative feedback. As you read this please keep in mind that if you don’t have a business need for blended methodology processes (and RUP is all you need) or if you are fine with having one or two process authors, RMC does meet your needs. In fact we have a handful of customers who are in such a situation and would simply like to move their RUP process into a collaborative development environment such as VSTS using Content Bridge for VSTS. I hope to see some improvements in the UI with RMC 7.2 and will share what I find out.

Software development governance is concerned with establishing standards and control mechanisms to enable practioners in a development organization to carry out their roles and responsibilities while building or maintaining software development programs. A software development governance framework should not only facilitate the creation of processes that may act as a guide for project teams based on accumulated best practices from experience, but it should also serve to ensure that essential procedures have been followed. Therefore any serious attempt to establish a sustainable software development governance framework needs to build or reinforce these three key pillars: establishing and declaring software development processes, automating the enforcement of governance processes, institutionalization of compliance best practices so that teams are empowered to take corrective action earlier rather than discovering problems later where penalties may result.

When we started researching the current literature and tooling to help us understand the current state of art we were unable to find a single resource that elaborated how to transition from theory to practice. This has made us start work on a whitepaper that would dig deeper while giving an implementation model for software development governance. In this paper we aim to describe how it is possible to deploy a software governance framework including its automated compliance using a collaborative development environment. We use Microsoft’s Visual Studio Team System (VSTS) to illustrate and give a practical recipe for achieving governance automation. After talking about setting up a governance process architecture we guide you through the steps needed to achieve a comprehensive, continuous detection and validation of process enactment and out-of-compliance events. To that end, we also plan to include an implementation reference model which can act as a base for your own specific implementation. By installing an automated governance infrastructure, preparing for internal and external audits requires fewer people and less effort.

I will surely blog about this again when the whitepaper is publicly available.

In recent years we have seen a trend of software companies, regardless of their size, realizing the benefits of having well defined software development processes in order to increase the success rate of their projects. One cannot argue that when it comes to software processes, one process does not fit all. With this understanding comes the challenge of how to ensure that an organization with a number of software development processes can ensure process selection is made properly.

At the very minimum, an organization needs to have a central place where consumers of the processes can view all available processes and search based on criteria such as process families (e.g. RUP) and process recommendations (e.g. team size). The richer the information included in this process hub, the easier it will be for interested individuals to find the appropriate process. By implementing this central place for processes, companies will see an increase in usage of their software development processes.

The next logical step is for an organization to promote its process center as a vehicle to improve processes through collaborative community input. Traditionally software processes are designed by elite group of highly skilled individuals, often in isolation. Without a feedback mechanism in place, there is no real chance for processes to evolve or survive. With the dynamic nature of software development projects and without the two way approach, processes quickly become obsolete and there will surely be resistance amongst project consumers to follow the processes. Organizations can easily over come this major issue by bringing together the process authors and consumers. Process centre can be a gateway for both parties to come together to share and collaborate on existing processes. The type of feedback can range from the most informal method such as rating system to most constructive such as recommendation on the specific process element or step. By having the wide range of options available, consumers can choose the method which suites them.

A natural way to implement this type of system is through the newly popular Web 2.0 technologies known as the second generation of web-based communities and collaboration tools. This trend in technology and industry addresses the need for collaborative way for the entire organization to work together towards evolving their software development processes and ultimately to increase the success rate of projects. I am pleased to say that we are currently working on a process portal based on Web 2.0 technology which is complimentory to IRIS Process Author. Stay tuned for more blogs regarding IRIS Process Central.

As soon as I hit ‘Publish’ on my previous post yesterday, I came across this very interesting announcement-
BEA Systems and Adobe Systems announced they will be partnering to provide Adobe’s Flex Builder 2 bundled with BEA’s Workshop Studio - this bundle due out later this year. On their part, Adobe will distribute BEA Weblogic Server evaluation instances with their LifeCycle Enterprise Suite (ES) offering - planned for early 2008.

An understanding is emerging that hosted application infrastructure- Web 2.0 or in the enterprise - cannot be considered complete anymore without inherent support for rich client application frameworks. SOA, web services etc. are enabling technologies but the workflow needs to include a sophisticated client that can leverage them - and vice versa.

Adobe’s also doing the right thing if you look at this from the perspective of competition. Microsoft’s Silverlight is already grounded on the .Net platform - With .Net 3.0 and it’s accompanying middleware technology set including WPF, WCF and WF- it now has a powerful foundation to lean on - not to mention a powerful development environment support in Visual Studio.
Adobe has it’s own middleware-like offerings (ColdFusion or JRun anyone?) but they are not nearly as ‘complete’ as the application servers such as Weblogic, IBM’s Websphere, JBoss and a few others. So they need this kind of bundling.

More on this announcement here.

I would describe a process management lifecycle as generally consisting of three broad phases. In the first phase a robust process architecture facilitates the tailoring and blending of various methodologies to come up with a process that is suitable for specific domain, category of projects or even a specific upcoming project. Methodology experts, subject matter experts, process engineers and project managers are the typical kind of roles that would contribute in this phase.

In the next phase, these processes are shared with a wider audience. This is essential to get feedback and increase process ownership within the project team that would have to ultimately enact it. A process portal would ensure that the processes tailored in the earlier phase are versioned, categorized and available for review. It is important that there is no overhead associated to consume these processes. You should make it convenient for a team of practioners (who may not be process experts) to locate, review and comment on the process they are interested in. The process portal would also be the single location from where users can download VSTS process templates, MS project templates, published online process and printable (document format) process. To encourage a transparent vibrant process community, tools like blogs, ratings, newsgroups should be used extensively.

Finally, in the third phase, the processes are now enacted in real projects. With the advent of ALM middleware like VSTS, this next generation process enactment takes the form where process can now be instrumented within the tools used during the project. This not only ensures a friction free process enactment but also ensures data-based decisions on process enhancements as well as compliancy auditing.

Those of you who are aware of the work done by Osellus in the past already know that we have tooling & services to support the first and last phases described above. IRIS Process Author and IRIS Process Live facilitate process authoring & architecture and process enactment respectively. Now we are getting ready to announce the work we are completing in supporting the process portal layer support. In fact within the next few days we will announce a process portal within Osellus.com that would provide the support for a community interested in process templates for publicly available and open source methodologies. Stay tuned.

Here is a post in the Eclipse Process Framework newsgroup that describes OpenUP as “a solution looking for a problem”:

“It may sound naive, but to me OpenUP looks like a solution looking for a problem.

There are already many excellent agile and formal methodologies out there. What development teams need is integration of these methodologies into tools they use daily, not another new methodology created by methodology high-priests. Jazz (Team Concert) is a great example of such a tool. It would be even better if IBM just open sources RUP, so we don’t spend any more time coming up with a new open source methodology. We may just have to accept the bitter reality that after two years (of marketing and promotion) there is not enough interest in OpenUp for a genuine community to form around it.”

If I tried, I couldn’t have said it better ! This is exactly what I had been reflecting on yesterday.

In the past I have pointed out the issues with scalability and lack of enactment (both in context of EPF and IBM’s RMC). However, this sentiment goes beyond and actually questions the need for “spinning the wheels” without adequate clarity on a direction to follow. Rather than wasting so much time, IBM could have been generous and open sourced RUP and not some percentage of it. Also it could have ensured that through a more thought out tool implemenation the historically high overhead related to process, could have been lowered and process “high priests” exposed. But for whatever reasons this was not done.

On Jazz, though I welcome this focus on collaborative development environment, first introduced a couple of years ago by Microsoft Visual Studio Team System, I have a feeling that this idea of “Open Commercial” license may end being a repeat of this half-hearted openness demonstrated by EPF.

What is a process architecture ? My last post talked about a project manager tailoring a process or flexibly enacting a tailored process in a project. In this post I would like to zoom out and present how a typical process architecture looks like. Later I would share the litmus test that shows if your process architecture has been created correctly.

In a typical enterprise organization, the top layer in a process architecture consists of process elements. These are elements like roles, work products, activities that go on to constitute a process. These elements could belong to one or more commercial frameworks or could belong to different methodologies or frameworks. These could also include constitutional elements which the organization needs to define and declare in order to meet a internal or external control objective as part of regulatory or quality compliance.

The next layer would consist of process components that are fragments or clusters of related activities. These process components make it convenient to concentrate on specific aspects of a process. Depending on the business or compliancy objectives of the organization you could craft these components to achieve a certain result or describe a desired workflow. One or more components could be linked to create a larger component and eventually an end-to-end process.

Below the process element and process component layer you will have the process layer. This is where you will have the various sub-organizations or departments start crafting their processes in tandem with the software process engineering group. You would not only use elements from more than one packages but in all probability you will be working with other process authors in different time zones or geographical locations. This process layer also ensures that rather than relying on one monolithic process pushed by a vendor or consultant, teams can create processes drawing from different methodologies and process frameworks.

The next layer would make it convenient for the project management office to tailor the process for a category of projects. This domain layer makes it convenient for you to tailor processes for a technology or business domain. Some common examples I have seen in this layer are tailored processes for web applications or those for mobile applications. This domain specific tailoring makes it convenient for project managers to easily search and use processes in real projects. This takes us to the last and final layer, the engagement layer.

The engagement layer consists of processes that are tailored for a specific project. The project manager does not have to go any higher than the domain layer to look for a process that she can specialize for a project that is about to be initiated. Project managers already have their plate full with all the project start-up tasks and they don’t want any overhead associated with searching for and tailoring a process for their project. They want a tactical intuitive way to tailor a process for a project. If they end up spending more time than needed to prepare a process it is likely that this would end up costing more than the benefits accrued. So the litmus test for a successful process architecture is the time taken by a project manager to tailor and operationalize a process in a real project. The lower this transaction cost of tailoring and moving a process to enactment the higher marks you get for a well managed process architecture.

Almost no discussion on managing processes is complete without the use of the word “tailoring”. It is one of those words that although is understood commonly at a generic level starts giving trouble when applied without enough explanation in process development scenarios. I will attempt to describe how I understand “tailoring” in context of process models. I will provide this explanation using a practical scenario of applying a process in real projects.

A process is of little use unless it is expected to add value resulting in successful delivery of real life projects. Every endeavour or project, be it a software project, science experiment or manufacturing an article of clothing, has a certain process it follows. For us in the knowledge intensive domain of software development, successful project delivery is made tougher due to changing or late arising requirements coupled with involvement of skilled knowledge workers. My earlier post talks a bit more on these challenges. Over time, an organization would accumulate a large number of lessons learned as well as patterns and practices that work consistently well across projects. They will also have a collection of anti-processes (think anti-patterns) which they know don’t work. Some organizations would use a tool such as IRIS Process Author to structure their body of process knowledge for easy reuse.

Let us now add a temporal element relating to a point of time when an expected endeavour/project becomes known. At this time, you would study the characteristics of the known execution environment and try to modify your existing body of process knowledge to come up with a process model that best suits this expected project environment. As much as you can, you would also account for the known unknowns in your process model. Based on different situations, this would mean defining things like - a set of activities and their work breakdown structure or workflow, a set of work products that are created and or consumed in the process, the process roles that are staffed by practioners, the pre-post conditions that might be true within the process and so on. You may also defined the narrative guidance that would be read by the practioners in the project. This tailoring of the available process body of knowledge will result in bespoke processes suitable for a specific project execution environment. This process model would best describe the process framework that is expected to result in a successful execution of the project.

Now we will add the second temporal element relating to a point of time when the project is initiated. At this time you would use the tailored process to create instances of process definition elements as project work items. For example this would mean a project manager creating instances of a change control workflow as and when a change work item is added. I would call this instantiation of process definition as flexible process enactment. Keep in mind that the project manager can also add additional work items that are not defined in the process model. However, these changes to the process model could not have been anticipated as the time of process tailoring before the project is initiated. Some would argue that even this flexible process enactment is a “tailoring” of sorts since the process is being instantiated based on the unfolding project (and in some cases new work items are added to the process model). Personally, I have no problems with that as long as there is clarity on the context – tailoring before project initiation and tailoring after project initiation - is understood.

The key point I want to underline is that you would not leave process tailoring to be done after the project has started. Rather you would attempt to tailor your process, using the lessons learned from the past and the known unknowns of the project, before the project is initiated. Once the project starts you would flexibly instantiate your process elements as project work items driven by the circumstances of the unfolding project. Off course you will also add the unknown unknowns to your process. However these by their very definition would not have been known before project startup so could not have been modeled ahead of time.

In my experience process authoring for software development processes is a highly collaborative activity. It is unnatural to expect a solo effort to come up with a process that could be successfully consumed by a wide team of practioners. The process infrastructure also needs to scale well. Reusing existing process components not only reduces significant overhead but also makes the task of coming up with customized processes agile. Any tool that attempts to support process modeling needs to address these two objectives. Based on the evidence I have noticed, the EPF Composer (and RMC from IBM) fails in this regard.

From the information available, it appears to me that there are significant problems being faced by the Open UP team as they try to come up with a process model. The fundamental issue stems from the fact the EPF Composer is designed as a single-user client tool over-relying on SCM to check-in/check-out complicated process plugins. In my opinion it is unrealistic to manage processes in this manner. You basically end up choosing one of two undesirable options. The first option is to break up a process into multiple constituent plugins that are managed separately in an SCM. The second option is to put everything under one library that is checked in. Here is an on-going EPF-dev discussion thread on this topic. This is a thread questioning the rationale for having a overblown library with every sort of plugin in it.

Most enterprise teams will probably face similar problems when using Rational Method Composer (RMC). If the OpenUP team is having difficulties dealing with the limitations of the tool when trying to add a single (DSDM) plugin to the base OpenUP methodology, imagine the magnitude of problem in an enterprise environment when a blended methodology process architecture consists of a number of methods, regulatory and compliancy standards. Moreover, in that environment there will be a large number of subject matter experts who would likely want to collaborate as they contribute to various aspects of the processes. With the difficulties I am seeing with the usage of EPF Composer within the EPF team one can imagine the problems being intensified with scale.

Providing a server side support for process management is critical for a scalable process architecture and collaborative development of processes. This was a fundamental business objective for the Osellus engineering team that came up with a robust database-backed enterprise architecture for IRIS Process Author. Since every process asset resides in a database, it is easy for IRIS Process Author users to discover, tailor and export processes as needed. All this is done without resorting to complicated check-in/check-out rules associated with handling processes managed as files.

Finally, some would say that it is ironic that the very team that is promoting the “eclipse way” is actually building products that seemingly takes one down the path of single client development rather than team-based development or worse still providing only limited visibility into source process content.