In the last few months. third party software development world has made a historical paradigm shift to cloud-based computing and App store distribution model.

The most significant aspect of this shift is the opening of almost any platform  (televsions, cars, netbooks, phones) to sophisticated third party applications. Each of these platforms has its own unique technology. Previously, these platform were only open to minute number of embedded developers closely working with the hardware vendors and the code was pre-installed on the hardware before it reach the consumer. Now any third party developer can develop apps for any number of these platforms.

Imagine developing an app  that users can download to their iPhones that interacts with an app that they have running on their Samsung HD TV. All through a cloud-based platform. The users simply go to Apple’s and Samsung’s App Stores and purchase the app. Or an app running on Ford cars interacting with Android phones. The possibilities are endless.

The multi-technology and app store distribution model of this new paradigm has significantly increased the complexity of the development process, hence making new generation of software development methodologies critical. For example, we have tailored some of our development processes to include micro iterations (to satisfy quick-time-market business requirement), and app store submission/approval activities.

Is it my imagination or there are short bursts of activity–usually initiated by IBM–followed by long periods of silence?  After so many years, why isn’t there a wider interest in EPF? Are we suffering from a “yet-another-methodology” overload?

This tired interest is an important indicator of how the user community feels about methodology fads.

This is the first post in a series detailing out a process we at Osellus have devised for development of iPhone applications. This process incorporates Apple’s best practices for iPhone development, as well as, the many relevant lessons learned from the general software development body of knowledge.

This work was initiated by our iPhone development and project management teams. As we embarked on various customer projects, the project management office wanted to have better project visibility and the ability to make sure the projects met the customers’ objectives. It  has also helped project managers with staffing of each project with the right skill set. We intend to improve this process based on lessons learned from the completed projects.

In the next post of the series, I will cover the overall business objectives, and the process objectives as they relate to each business objective which thereby form the basis of the process.

Software development processes capture—in reusable form–the organization’s best practices and lesson learned, making them sharable across projects. Today, the benefits of process-centric software delivery are well understood. So, why is the industry’s adoption of software development processes so dismal? The lower than expected adoption can be partly explained by a phenomenon called groupthink. For this post I will rely on materials from James Surowiecki widely cited book: The Wisdom of Crowds.

The coinage of the term, groupthink, predates Surowiecki’s work, but he frames it concisely within the larger arena of group decision making, which he shows is more accurate, in most cases, rather than decisions made by subject matter experts. This easy to read book draws from and consolidates various scientific and empirical bodies of work from diverse fields–such as psychology, statistics, and economics–making the subject generally accessible.

For crowds to produce correct decisions, its members must be diverse, independent, and decentralized, and should have a mechanism to consolidate the individual judgments into collective decision. However, the decision making fails when the members of the crowd are too conscious of the opinions of others and begin to emulate each other and conform than think differently. This failure is called groupthink.

I believe commercial software methodologies have been suffering from groupthink. For over a decade, most efforts have centered around Unified Process with all participants—mainly methodology theorist and consultants-emulating each other and conforming rather than thinking differently. Any new development—such as Eclipse Process Framework or SCRUM—has been forced to fit in a UP mold. The practitioners have found these expertly devised methodologies irrelevant, and, hence, have mostly avoided them. At the same time, new practical ideas that arise during actual development projects are prevented from blossoming. The potential methods devised by diverse projects’ practitioners are likely to be more relevant, as they convey the wisdom of crowds (mobs) and consequently have a better chance of wide adoption.

The good news is that with the recent availability of integrated ALM and interactive process asset repository systems, it is now possible to involve practitioners in the end-to-end methodology development effort. I will cover this in more details shortly.

As if the development of software systems was not complex enough. It is about to get even more complex.

In the past few years, there have been much fruitful activities in the area of programming languages resulting in introduction of various languages. Although some of these languages have been around for a number of years, the recent availability of their robust, high-performance versions has made their usage in large commercial systems possible. Broadly speaking, these new languages can be grouped into two categories: dynamic languages and functional languages.

Dynamic Languages

For many developers strong-typed languages are too restrictive, contextually unintuitive, and verbose. Dynamic typed languages, on the other hand, are simple and elegant, ideally suited for rapid prototyping and development. They are specially well suited for time-sensitive development projects where time-to-market is critical.

Functional Languages

With the advent of universal deployment of multi-core hardware and the eminent availability of cheap massively parallel systems, the need for making concurrent programming mainstream has attracted much attention. The non-imperative aspect of functional programming results in code with no side-effects, well suited for concurrent programming. Functional programming is also very efficient for implementation of certain algorithms resulting in dramatic code reduction.

Multi-language Platforms

Aside from robustness, the new versions of the above languages are design to run on common runtime environments: virtual machines. This allows co-existence of multiple languages in a single software system.

Imperative type-safe object-oriented languages—such as java and C#–dynamic languages—such as Groovy and IronRuby—and non-inperative functional languages—such as Clojure and F#–can run seamlessly together on commercially available virtual machines–such as jvm and .net clr/dlr. Therefore, no technical barriers to development of multi-language systems exist today.

Example: Mobile Application with Cloud-based business logic.

Imagine a mobile application that provides sophisticated investors with instantaneous alerts based on news on or coverage of a company. Such an application constantly scans large number of online sources for any mention of the company’s name or reference to its competitors or general macro-economic events that can affect it. The system, then, analyzes the raw inputs and provides the user with a single metric indicating the effect of the latest chatter on the company’s stock price (positive or negative). Based on this metric, the investor can take appropriate actions.

Development of this system requires both mobile client and server side development. The client side could be an iPhone application implemented in Objective-C and UIKit.

The server side is more complex. It requires a concurrent component running on a massively parallel cloud computing platform that constantly monitors and filters large number news feeds, blogs, and possibly social networking sites. The analysis of the filtered information may require specialized algorithms some aspects of which require constant enhancements. These functionalities are best implanted using combination of object-oriented, dynamic and function programming languages

Process and Multi-language Development

Because of the inherent complexity of multi-language systems, software development processes and integrated application lifecycle management (ALM) systems are critical to development of such systems. Imagine the various categories of practitioners and skills involved. Each with its own unique development, validation and verification approach and tools.

Earlier this month, IBM Rational announced their Measured Capability Improvement Framework (MCIF) offering, vaguely described in the IBM’s oficial news release:

Additionally, with IBM’s new Measured Capability Improvement Framework (MCIF), organizations can also take actions to continuously improve on results by learning from past experiences. Through MCIF, IBM provides organizations with an end-to-end framework that enables them to measure results and manage projects so they can incrementally improve their software delivery capability. 

“In today’s economic climate, businesses are looking for new ways to derive greater value from their investments in software,” said Dr. Daniel Sabbah, general manager, IBM Rational Software. “Up until this point, organizations have been lax in measuring the business value and discipline of the processes [emphasis added] they use to deliver software assets. Classic metrics in software engineering largely ignore the importance of actual business outcomes. Our clients are now beginning to realize that the software they build or assemble must be treated as a strategic business asset. IBM is committed to helping them make the right decisions and improve the successful outcomes of this newly emerging business process discipline.” 

 

Measurement driven improvement is central to CMMI and CobiT has a strong emphasis on alignment of IT processes with organization’s business objectives. If you have read my blogs, you would know that I strongly believe in objective-based process definition. If anything, IBM Rational is playing catch up. But it is still nice to have their confirmation.

I didn’t attend this years Rational Software Conference, but I have been carefully studying white papers on MCIP and Rational Insight offerings. I will discuss Rational Insight in a future post.

MCIP white paper is well written and is an enjoyable read. I fully agree with the framing of the differences between business and manufacturing processes and software development processes on pages 3 and 4.

Unlike most other business processes, such as supply chain management or manufacturing, SSD needs to deal with a range of risk. SSD also differs from many other business processes in that it entails a diseconomy of scale: that is, individual productivity decreases with the size of the SSD effort. …

Software delivery differs from many other business processes by dealing with a broad range of innovation. Some software projects, such as maintenance of existing systems, are reasonably predictable, similar to manufacturing processes. Those projects carry limited innovation and drive limited or no business differentiation. Other projects, such as building unprecedented and large software systems, require high degrees of innovation in addressing problems that have never been solved before on a schedule. Committing to delivering innovation requires assuming risk, since the lack of complete knowledge at project inception is inevitable and uncertainty regarding how to proceed is part of the challenge. This risk is manifested in the statistical variance in the estimate of the time or cost to complete. 

A commitment to assuming risk entailed by bringing innovation to the enterprise provides the opportunity to improve ROI.  

Another major difference between the business process of software delivery and other business processes is the diseconomy of scale. Typically, manufacturing and service delivery processes offer economy of scale: The cost of a unit of software grows nonlinearly (i.e., yields cost reduction) with the size and complexity of the system. But this is not the norm in software production.

On the other hand, some of the insights that have been discovered as part of IBM effort are trivial. For example, on page 15 they say 

Many organizations mistakenly try to make one process fit all circumstances. In our experience, the above type of analysis is required to enable you to drive the appropriate change to the right project types. 

I don’t know of any organization that doesn’t believe this. In fact, it sounds condescending. 

In essence, MCIF is a practice-based approach to software development processes. An approach they first introduced in the last version of EPF (before it became inactive). One can argue that IBM was a later comer to this also., The concept of practice has been widely utilized in CMMI, Microsoft MSF and EssentialUP.  MCIF is a methodology for top-down selection of practices based on the organization’s business objectives.

Although I like objective-based software development process definition, MCIF, however, is top-down and non-collaborative. It relies on Rational Method Composer (RMC) tool, which is a single-user desktop application–requiring a configuration management system for basic maintenance of processes. The white paper, also, falls short in addressing the practical issues of mapping business objectives to different aspects of processes and the mechanics of process tailoring. 

Finally, from Per’s video, it is apparent that MCIF is not a tool empowering users, rather it is a service that requires engagement of IBM consulting services.

My recommendation: best source for software development capabilities improvement is CMMI body of work. As I said before, CMMI is the result of two decades worth of work by various subject matter experts, not a single vendor’s commercial methodology.

A colleague forwarded to me an interesting work by Petter Holmström titled “Ideas for Next Generation Process Authoring Tools”.  It’s a long comprehensive document, and I have just started reading it end-to-end. From a quick scan of the table of contents, abstract and conclusions, I mostly agree with his conclusions and recommendations:  

The tool vendors should shift focus and concentrate on making their tools more collaborative, customizable and scalable to different process sizes. In this thesis, some ideas of how this could be achieved have been presented, of which one of the more interesting ones is a wiki-based authoring tool.

 

As you may have realized from my previous blog postings, I am a strong proponent of collaborative process management tools and the importance of the involvement of developers and other process consumers in the creation of processes–they consume. The industry players and the user community should democratize process authoring and move on from blindly following methodology pundits.

IT and software development processes and business processes are inherently different and are driven by their own domain needs. System workflow and service oriented architecture (SOA) are the main drivers of business processes. Low-fidelity human centric activities are the main drivers of IT and software development processes.

Execution of inline machine logic and invocation of external systems form the majority of business process logic; human interactions are limited to simple tasks such as approvals.

In contrast, low-fidelity activities of IT and software development processes only capture, at high-level, the expected work to be performed. These processes also rely heavy on role-based creation of work products—for example system analysts create requirement documents, architects create design documents, tester create test cases, and developers produce code. Also, the workflow aspect of these processes is indeterminist and it heavily relies on the knowledge workers involved.

Based on these fundamental differences a single set of systems cannot satisfy the needs of both IT and software development processes and business processes.

IBM Rational team launched EPF more than five years ago with a premise of creating an open source like community for software development processes and methodologies. Unfortunately, from the beginning, the Rational team was not willing to give up control and EPF has been only open in name than in reality. I would say, 99% of the contribution so far has been from IBM. Initially EPF was supposed to be a kernel based methodology. Plug-ins were to be added to the kernel as to satisfy the requirements of development project environments. This takes an additive adoption of methodology rather than RUP’s trimming approach. After few years of taking this approach, suddenly EPF changed direction and took a practice based approach. Now it seems that the IBM Rational team has abandoned EPF and has moved to their latest story: Measured Capability Improvement Framework (MCIF). I don’t know much about MCIF right now, but intent to research it.

My point is that this is an irresponsible behavior by respected methodologies, it confuses the potential adaptors and the grater market. I rather have IBM methodologies (content team) keep a low profile and let the badly need Jazz (very impressive IBM tool to rival VSTS) based tools flourish.

These days we have a methodology overload.  It’s not that I don’t recognize the importance of SDLC methodologies, rather I feel we already have more than enough methodologies to satisfy the needs of any organization. My proposal: lets put methodology development on pause for two years.  During this grace period everyone should focus on development and deployment of tools that support existing methodologies and processes. These tools should codify any existing methodology and simplify its application in real-world projects. In short, lets give what organizations have planted a chance to grow so it bear  fruits.

Just consider the wide variety of methodologies available today: SCRUM, Extreme Programming and other agile methodologies, MSF, Eclipse Way, RUP, EssentialUP, OpenUP, double-Chocalte-PestasioUP (and other flavor of the day UPs).

This methodology overload has resulted in confusion and  paralysis of many organizations trying to adopt and deploy processes in their software development projects.

As a final note, I would be grateful if methodology theories put their book authorship careers on hold for two years for the the betterment of the end users.