One of the challenges in adopting any methodology in an enterprise is in ascertaining the degree to which it is actually used. Although this sounds simplistic enough it brings a key issue out in the open. Even if an organization has converged on a single methodology to use there are subjective and often acrimonious debates on the extent to which a defined process adheres to the key concepts – either fundamental concepts that outline the methodology or the control objectives that have been bolted on to this methodology. It is not enough to have a methodology preamble announcing that this methodology meets so and so concept or follows this philosophy. A process mature organization needs to go behind the rhetoric (and I am not trying to slam this sort of collateral that sells the methodology) and actually give specific mapping between the process elements and the concepts and or control objectives that are promised to be met with this process.

This data-mapping will not only help in auditing your processes for compliance to your business or regulatory objectives but will also serve as an indisputable baseline to have those debates on what kind of process does the project team really need for an upcoming project. If you know where you are now, it is easier to steer where you want to go. Categorization of your process elements is therefore a simple but powerful way to keep measuring if your process is on track to become what you expected it to be or of you need to make modifications so that you can control its flavor at process authoring stage. Off course another benefit in this approach is that when you are ready to share your process with a peer network they can give their feedback so that you can adjust the mapping of your process elements to the methodology concepts and control objectives and validate this mapping.

There are many examples of this sort of categorization that can be seen in the processes available at the IRIS Process Central Sandbox. You are welcome to try out these processes and provide your comments.

I have been keenly watching the progress on what started as a promising project in the Eclipse foundation. Unfortunately after more than two years of opportunity to shine I would have to give the EPF project a failing grade. I fear that when someone at IBM realizes the wasteful investment of resources in this project they will pull the plug on their participation at EPF. This will inevitably kill the project since IBM is the only committer on the tool aspect of EPF and the major committer and donor to the content aspect of EPF.

In my view there are several reasons for this failure and many lessons that I have learnt simply by following the travails of the project. These reasons are not the lack of aptitude and personal contribution of the EPF team but simply a miscalculated approach to building a vibrant self-organizing process community. Today I will highlight the mind boggling high transaction costs of submitting process content to be consumed and critiqued by a peer network. Even by their own admission, it takes “2-3 months” for the eclipse legal process to approve any process contributions to EPF. This is an anachronistic concept in today’s day and age. Communities are self-organizing. If what I contribute is useless to you, you will not use it. Period. Moreover having been tagged as someone who supplies useless content my user ratings will go down and I will have to significantly improve my game to get back my reputation. Communities give rise to wisdom-of-the-crowd, encourage meritocracy and do not tolerate back room machinations. As articulated by Chris Anderson in The Long Tail while discussing the tools needed to support a long tail, “This is the end of spoon-fed orthodoxy and infallible institutions, and the rise of messy mosaics of information that require - and reward – investigation”. Elsewhere in the book he discusses how “a once-monolithic industry structure where professionals produced and amateurs consumed is now a two-way marketplace, where anyone can be in any camp at any time. This is just a hint of the sort of profound change that the democratized tools of production and distribution can foster”. In the case of software process modeling, these tools take the form of easy to use process modeling and consumption tools. With that, rather than a paradigm of a handful of processes being donated by a few process experts, one could feasibly have hundreds of process variants created, put on wikis, blogged and rated! Now if we multiply the 2-3 months timeframe for these 100 processes the EPF legal staff will be kept busy for 20 years…just on this project! Later I will discuss some other pitfalls to avoid based on the lessons learned from following this project.

Yesterday Omid and I gave a joint Microsoft-Osellus webinar on Effective Software Development Governance with Adam Galant. Thanks to Rini Gahir and Adam Galant from Microsoft Canada for arranging this webinar.

It was a challenge to fit an explanation of the theoretical concepts of SD Governance with a tangible step-by-step approach to achieving it within the 45 minutes we had. In the end we were able to keep to our time and I hope the audience (about 36 attendees) left with an outline of a practical approach to establishing software development governance in an automated yet flexible manner.

Here is a link to a recording of the governance webinar.

You can use this link to register for the joint Microsoft-Osellus Webinar on software development governance. Omid is going to be talking about a lot of things that we have been able to achieve with the collaboration between the osellus process toolset/service offerings and the VSTS integrated ALM.

Here is a preview of what we will be talking about:

Effective governance aligns your software development investment with your business strategy. By ensuring continuous traceability of software development activities, governance leads to enhanced customer satisfaction and compliance with internal and external policies.

While most organizations use some measure of implicit governance, the full potential of a governance framework can be realized by taking three practical and explicit measures. The software development processes in the organization need to be selected and tailored in a manner that supports organization’s business goals. These processes need to be enforced in a streamlined manner. And, they need to be monitored to ensure corrective actions are taken in a timely manner.

In this webinar we outline a practical approach to establish software development governance in an automated yet flexible manner. We will discuss how the business objectives, standards and control mechanisms in the organization should be mapped to process elements such a roles, activities and work products that guide practitioners in a non-intrusive manner. Microsoft Visual Studio Team System (VSTS) and the IRIS process automation suite from Osellus will be used to demonstrate implementation of an effective governance framework in real-world scenarios.

In September I had talked about the ongoing work the Osellus engineering team was doing to address the enterprise need of establishing a sustainable process community.

I had mentioned how:
“This is essential to get feedback and increase process ownership within the project team that would have to ultimately enact it. A process portal would ensure that the processes tailored in the earlier phase are versioned, categorized and available for review. It is important that there is no overhead associated to consume these processes. You should make it convenient for a team of practioners (who may not be process experts) to locate, review and comment on the process they are interested in. The process portal would also be the single location from where users can download VSTS process templates, MS project templates, published online process and printable (document format) process. To encourage a transparent vibrant process community, tools like blogs, ratings, newsgroups should be used extensively.”

Well, Negin has announced the availability of the IRIS Process Central sandbox that introduces this solution.

In my previous posts I had touched upon an important challenge facing Microsoft and IBM as they introduce “game-changing” integrated ALM offerings. How does one go about convincing IT decision makers to start adopting this ALM infrastructure. Outreach activities and a thriving partner eco-system is one of the ways to get the idea out in the open. One example of this is the Jazz team’s attempt to showcase their offerings at the prestigious CASCON 07. This collection of researchers and academics is a fertile seeding ground for advancing the state of art in these concepts.

Yesterday, I attended a BOF session by Marcellus Mindel that discussed opportunities for using Jazz in an academic setting. There was a good discussion on the challenges instructors face in deploying and using Jazz in a curriculum. One of the questions was on curriculum pre-requisites on using Jazz. Do you use Jazz in a first year course before students are aware of the software engineering processes and practices or do you introduce it in fourth year when they have had a chance to get a grounding on some of these practices in addition to essential technical knowhow? Integrated ALM tools like VSTS and Jazz can be an excellent project simulators where instructors can use different process templates to give challenging assignments (a lot like using case studies). Of course, someone may choose to use the ALM tools simply to encourage collaboration between students as they work on their assignments. There were other discussions around research into areas relating to process templates, reporting and generating best practices by using enactment data – all of the concepts that I am passionate about and have blogged in the past.

Later we had a chance to visit the exhibitor area and chat with the IBM Rational Team Concert folks from the Toronto Lab. For those who don’t know, IBM’s Toronto Lab is one of the important brain trusts collaborating on the Jazz/Team Concert platform/product. We wish them our best as they continue to ramp up their outreach activities and aim for the general availability of IBM Rational Team Concert sometime middle of next year.

Here is the Osellus team at the Jazz booth - notice we are appropriately geared !

From left to right - Dave, Payman, Me, Kartik (IBM), Rakesh, Jim, Jason

Last night I had a chance to attend a Jazz demonstration by John Wiegand at CASCON 07. It is always a treat to watch John speak and yesterday’s session was no different. For those who have not seen John and Erich speak about collaborative development, here is a short video to one of their previous videos talking about open source development.

John gave a glimpse of the current state of Jazz technology. Although Erich was unable to join the demonstration concurrently from OOPSLA in Montreal as planned, judging from the reaction of the standing room only audience the session was well received. There are several cool things that were shown including the concept of personal work spaces that amongst other things let developers see the effects of checkins before touching the main code repository. John ran through a scenario in which two new developers join an existing project. The system configured the tool behavior for these newcomers based on the process definition. Living up to the promise of transparency, these newcomer developers get access to all work items in their project and across different projects. They then get down to the brass tracks by working on their own work items while engaging in adhoc collaboration and sharing changes with others in the team.

I had a chance to chat with John after the demonstration. Since my focus was primarily on understanding the process awareness built into this integrated toolset, John was quick to point out their definition of process in Jazz. Process in Jazz means the policy and configuration settings that govern tool behavior. The process definition such as checkin polices and build management can be customized for individual projects. We then had a candid discussion on the need to have balance between having a process definition that spawns project work items without burdening developers with complicated process terminology. I can’t go onto all the details here but we agreed that over-complicated process models (based on over-engineered meta-models that require a steep learning curve) could be harmful to the concept of collaborative development environments. This is the reason the Jazz team wants to be very careful when they talk about process and their definition of it.

John was also generous to help me with a favour. He has promised to see how can he make the EclipseWay available to the open source community. This will mean anyone can take this collection of best practices and mash them up with their own in-house process description or third party process definitions without getting a knock on their door by a team of IBM lawyers. I know many people in the EPF community had been asking for such a thing and will join me in thanking John for helping us with this.

I am back at CASCON on Thursday to join a BOF session on Jazz.

Next week will find me and some other members of my team attending the Jazz sessions at Cascon 07 here in Toronto.

As I had talked about in my earlier blog post, I am really pleased to see the work Erich’s team has been doing on the jazz platform. Moreover it is nice to see so much Canadian talent working on developing this next generation ALM platform!

From what I keep hearing, the Jazz platform and its first product manifestation in IBM Rational Team Concert is sounding more and more like Microsoft’s Visual Studio Team System. This is a good thing and something to take note. In fact I would even term the release of Jazz and VSTS as “game-changing” events that have the potential to alter the state of software development lifecycle tooling. This would also be giving sleepless nights to those vendors who claim to offer traditional ALM tooling consisting of a collection of development tools. Simply put the traditional ad-hoc installation and configuration of a collection of individual applications has just not worked. Moreover, customers were expected to go through an expensive and prolong point-to-point integration of these tool “silos”. Even the most comprehensive customer deployment only resulted in partial integration and never met the promised results. Do I need even need to go further on this point…

The key distinguishing characteristic of these ALM offerings from Microsoft and IBM Jazz is the infrastructure layer. Finding similarities with the application middleware (another game changing event) Payman, calls this layer the “middleware” that glues tools and processes together. The infrastructure layer facilitates concurrent development projects each with its own team and tools. It’s a common repository of projects’ data. Unified management of project data makes governance, visibility, predictability, traceability and other ALM business benefits possible. At the core of the infrastructure layer is a unifying object model. The object model is accessible via a comprehensive set of APIs. The ALM tools utilize these event-based API’s for deep integration with the infrastructure layer. Since these APIs are open and published, any third-party tool can also integrate deeply into a ALM solution. Each tool may have its own object model but as part of integrations the tool’s object model is mapped to the infrastructure object model.

The common infrastructure object model is based on the key concept of work items. Microsoft has a few years lead in introducing this concept by way of Visual Studio Team System. Work items are project specific elements, created and updated throughout the life of the project manually or automatically by tools and processes (based on defined rules and constraints). A work item is an instance of a work item type, which is defined as part of the configuration of the ALM solution. Work item types can be associated with each other by user definable link types. Therefore, the object model is fully configurable which makes the ALM solution very versatile. Additionally, the infrastructure layer provides a powerful reporting and data mining service, supporting user defined dashboards and reports for an individual project or consolidation of multiple projects.

This opens the field for tremendous innovation. A community of plug-in providers can now add missing or extend existing functionality. There is enough room to identify and build competencies on this platform - both product and service opportunities. This means a more thriving community that ultimately results in a better ROI for enterprise customers. But there is still a big challenge left. Remembering that they have heard these “cry wolf” calls on ALM promise before, how can IT decision makers be convinced that this is the real deal!

I will share if I hear something interesting at the Jazz sessions at Cascon 07.

In my view IBM is struggling to align their products and practices with the advancement in the state of around software development process automation and collaborative development environment.

In this post I am going to focus on their release of Rational Method Composer 7.2 (RMC 7.2) and specifically point out why it is difficult for me to be convinced that they have done a root cause analysis of their previous failures (with RPW or earlier versions of RMC) or have recognized the direction where software process automation is headed.

When one thinks of a product release from IBM you would obviously expect a high level of maturity, sophistication and thought leadership. IBM is universally well regarded and respected for pushing concepts like SOA and software-as-service. However, when you look at how the Rational team has implemented the tooling around creating and tailoring processes, one wonders why that thought leadership is not being put in practice here. By choosing to keep RMC 7.2 as a thick client application the rational team seems to be stuck in a time warp. I simply cannot imagine a scenario where an enterprise is told that its process assets (which in many cases have the “secret sauce” of project success) is encased in a few laptops. So I was bit surprised that even while acknowledging this problem of scalability and multi-methodology blending, the rational team has chosen to resolve it by continuing to stick to the flawed design decision of using SCM to check-in/check-out process plug-ins. But now, to make matters even more difficult, process authors (often non-programmers), are required to use the eclipse IDE environment to derive some scalability benefits. This is simply unbelievable. First of all, this still does not solve the problem of scalability (see why). Secondly, SCM is a code repository specially built for handling code and not process assets. Finally there is little logic in making process authoring so complicated that it requires a team of consultants to even setup the multi-layer environment for process authoring. Moreover in this day and age when even videos are being edited online asking a process team to work in this client-only disconnected mode is simply irresponsible. I did notice that the rational team did take note of the overwhelming negative feedback on usability of RMC and I welcome that. As in IBM partner and while exhibiting at the RSDC in June 07 we heard lot of this negative feedback. As you read this please keep in mind that if you don’t have a business need for blended methodology processes (and RUP is all you need) or if you are fine with having one or two process authors, RMC does meet your needs. In fact we have a handful of customers who are in such a situation and would simply like to move their RUP process into a collaborative development environment such as VSTS using Content Bridge for VSTS. I hope to see some improvements in the UI with RMC 7.2 and will share what I find out.

Software development governance is concerned with establishing standards and control mechanisms to enable practioners in a development organization to carry out their roles and responsibilities while building or maintaining software development programs. A software development governance framework should not only facilitate the creation of processes that may act as a guide for project teams based on accumulated best practices from experience, but it should also serve to ensure that essential procedures have been followed. Therefore any serious attempt to establish a sustainable software development governance framework needs to build or reinforce these three key pillars: establishing and declaring software development processes, automating the enforcement of governance processes, institutionalization of compliance best practices so that teams are empowered to take corrective action earlier rather than discovering problems later where penalties may result.

When we started researching the current literature and tooling to help us understand the current state of art we were unable to find a single resource that elaborated how to transition from theory to practice. This has made us start work on a whitepaper that would dig deeper while giving an implementation model for software development governance. In this paper we aim to describe how it is possible to deploy a software governance framework including its automated compliance using a collaborative development environment. We use Microsoft’s Visual Studio Team System (VSTS) to illustrate and give a practical recipe for achieving governance automation. After talking about setting up a governance process architecture we guide you through the steps needed to achieve a comprehensive, continuous detection and validation of process enactment and out-of-compliance events. To that end, we also plan to include an implementation reference model which can act as a base for your own specific implementation. By installing an automated governance infrastructure, preparing for internal and external audits requires fewer people and less effort.

I will surely blog about this again when the whitepaper is publicly available.