We have been working on a lot of apps on mobile platforms, especially the iPhone and as I mentioned in my previous post, we keep coming back to the problem of data integrity – in the cases where there is a server component and data needs to be synced up with the server.

Apps that depend on data on a server have several options of accessing data on servers. There is the ubiquitous RSS – get and read feeds. This is great for reading data off a server but quickly loses relevance when the application has a more complex data model and the app needs to write data to the server. The problem of writing data to server is easily solved through web-services, REST services and the likes through which you can talk to a service that takes care of translating your data into the server’s database.

The problem however is keeping the data on the device in sync with the server component. This problem is prominent on mobile devices because while your app on the device is working and live, there is not guarantee of continuous sustained connectivity with the server. Network connectivity is unpredictable and unavailable based on where you are, there are other problems that may interrupt your application. For example on the iPhone, an incoming phone call may interrupt your app while the data is being synced with the server. Your app or your device may crash.

Given all these possible scenarios, how do we ensure that data on the device is always in sync with the data on the server and secondly how do we ensure the integrity of the data given the number of breakages and opportunities for data corruption given the very plausible scenarios I mentioned above.

Over the next few weeks, as we evolve and implement our solutions to this problem, I plan to write a series of posts that looks at this problem in depth and possible solutions and their feasibility. One of the obvious first things we looked at was what are apps out there currently doing to address this issue of data integrity? Answer? Surprisingly little. Most apps out there simply don’t allow the user to do anything that requires server connectivity if the apps detect that there is no connectivity. This is not a great solution, and is primitive – especially given what we have come to expect on the desktop. Solutions like Google gears and soon the offline support in HTML5 make what’s being done on the mobile platforms look primitive.

So, why are we looking at this problem now? Of course we are motivated by the fact that so many apps face this problem but there is no well defined solution and doesn’t look like anybody is looking at it. But perhaps more importantly, technologically the tools are in place at this moment in mobile platforms for us to come up with a solution. The tools and utensils are there, the ingredients are all laid out for us, nobody has come up with a recipe yet or even entered the kitchen.

The iPhone, the Android and soon the Blackberry with v5 of the Blackberry development platform have support for SQLite – a full fledged transactional light weight embeddable database engine on the device itself. The mobile platforms themselves have powerful development kits that allow us to come up with robust solutions in combination with SQLite (and other mobile database engines).  The tools are there.

Over the next few weeks, I am going to describe various possible solutions and think through their feasibility and suitability through my posts. I think that would be the best way to get peer feedback and ideas as we hardly know everything there is to know and your input would be invaluable.

Keep an eye on this space.

It’s been around 2 months since my last post! We have been very busy with some very interesting projects. This year has had a good start from the point of view of getting to work with some very new and cutting edge implementations.

My team just finished work on a custom design/modeling tool for Yardistry implemented in Papervision3D and Flex 3. I have not seen something so functional and complex implemented in 3D on the browser platform. I am pretty proud of the work the guys put into building this product both in terms of technology and usability. I really encourage you to go play with it and use it once it is out at http://designer.yardistrystructures.com/. The custom design tool should be going live there within the next few days. I’ll put out a post once it is live.

The other thing I have playing with a lot in the last few weeks is the iPhone development platform. The popularity of the platform just cannot be ignored and Payman, I and  a few others have been putting a lot of thought into leveraging this first really usable mobile platform to bring our ideas to life.

Every serious developer has looked at some mobile device or platform in the past few years and had a few ideas for it but been frustrated with the sheer unfriendliness of the platform both for the user and the developer. I think the iPhone has changed all that – it’s a very serious mobile computing platform.

Payman put out a CMMI Reference App very recently and you are going to see many more in the next few months. The CMMI app is free. I know a lot of you come from the process engineering background and might find it very useful. Please don’t forget to rate it and leave a comment! Here’s the direct link to it in the app store: Click here.

I have been working on a very exciting iPhone app myself. I hate to be a tease but I cannot talk about specifics. Lets just say that it leverages social networks like Facebook and twitter to bring something very practical and functional to a crucial experience in a person’s life. Sorry, but that’s about as specific as I can get. I wanted to bring it up so that I can talk about a couple of things I have been thinking about since I started working on the app.

If you are an iPhone user, you probably groaned when you saw the words “social network”, “Facebook” and “Twitter” in reference to an iPhone app. I understand. There were an estimated 65,000 third party apps in the App Store mid July and it sometimes look like all of them deal with social networks.

We understood this. However, there is a difference between focusing on social networks like most apps tend to do and what we are trying to do – which is leverage them with a lot of attention to providing users with the benefits of integration with their favourite social networking sites without becoming about the social networks.

What we are realizing is that in order to differentiate between our app and the other thousands of apps in the app store, we have to do two things well.

First, it is no longer enough to have an application that has no server component. If your app is dependent solely on third party sites like Facebook, Twitter etc, you are always bound tightly to their APIs, restrictions and performance. Of course you are getting a lot - like a ready made audience and functionality that you don’t want to re-invent. But remember that everybody else out there is also using the same APIs and they probably work exactly like your application.  You have to enhance that experience in some way. The best way to do that is to have your own custom server component that adds some richness to the experience. You can layer this server component on top of existing social network sites ideally or add it to the list in order to achieve differentiation. The server component may or may not have a Web UI but it probably will have web services that cater to your app specifically.

Secondly, the integration with social sites should be balanced so that a user’s audience or their friends on those sites are made aware of your activities through your app but the app itself should not become focused around the third party sites.

A good example that comes close to illustrating my points is the GAP StyleMixer iPhone app. The StyleMixer app allows you to create and style your own outfit on the iPhone. I like the way it handles sharing the outfits a user creates. It allows you to share your outfits with your Facebook friends but also to GAP’s own StyleMixer community.

By doing that it gives you your standard Facebook experience but also gives you a specialized GAP community experience that would not be possible inside Facebook. It also means that the focus is around the StyleMixer community. Support for  third party sites like Facebook, Twitter can be added, removed based on their popularity. Your app is not very dependant on them anymore and works better because of the specialization you have built using your own server component along with a hook into Facebook and twitter.

Perhaps some of what I say above flies in the face of current practices but if you are building a serious product, I would say that the ‘Cloud’ becomes a rich back-end for your app only if you have a say in it – and that means your own server component more often than not.

It is easy to think that a server component for a mobile application is over-kill. But thanks to the iPhone and potentially platforms like Android, the time has come where a mobile application  is serious business and can be the backbone of your enterprise. If companies are getting funded for their iPhone app ideas, then we better believe its time to consider building more than a client only experience.

As if the development of software systems was not complex enough. It is about to get even more complex.

In the past few years, there have been much fruitful activities in the area of programming languages resulting in introduction of various languages. Although some of these languages have been around for a number of years, the recent availability of their robust, high-performance versions has made their usage in large commercial systems possible. Broadly speaking, these new languages can be grouped into two categories: dynamic languages and functional languages.

Dynamic Languages

For many developers strong-typed languages are too restrictive, contextually unintuitive, and verbose. Dynamic typed languages, on the other hand, are simple and elegant, ideally suited for rapid prototyping and development. They are specially well suited for time-sensitive development projects where time-to-market is critical.

Functional Languages

With the advent of universal deployment of multi-core hardware and the eminent availability of cheap massively parallel systems, the need for making concurrent programming mainstream has attracted much attention. The non-imperative aspect of functional programming results in code with no side-effects, well suited for concurrent programming. Functional programming is also very efficient for implementation of certain algorithms resulting in dramatic code reduction.

Multi-language Platforms

Aside from robustness, the new versions of the above languages are design to run on common runtime environments: virtual machines. This allows co-existence of multiple languages in a single software system.

Imperative type-safe object-oriented languages—such as java and C#–dynamic languages—such as Groovy and IronRuby—and non-inperative functional languages—such as Clojure and F#–can run seamlessly together on commercially available virtual machines–such as jvm and .net clr/dlr. Therefore, no technical barriers to development of multi-language systems exist today.

Example: Mobile Application with Cloud-based business logic.

Imagine a mobile application that provides sophisticated investors with instantaneous alerts based on news on or coverage of a company. Such an application constantly scans large number of online sources for any mention of the company’s name or reference to its competitors or general macro-economic events that can affect it. The system, then, analyzes the raw inputs and provides the user with a single metric indicating the effect of the latest chatter on the company’s stock price (positive or negative). Based on this metric, the investor can take appropriate actions.

Development of this system requires both mobile client and server side development. The client side could be an iPhone application implemented in Objective-C and UIKit.

The server side is more complex. It requires a concurrent component running on a massively parallel cloud computing platform that constantly monitors and filters large number news feeds, blogs, and possibly social networking sites. The analysis of the filtered information may require specialized algorithms some aspects of which require constant enhancements. These functionalities are best implanted using combination of object-oriented, dynamic and function programming languages

Process and Multi-language Development

Because of the inherent complexity of multi-language systems, software development processes and integrated application lifecycle management (ALM) systems are critical to development of such systems. Imagine the various categories of practitioners and skills involved. Each with its own unique development, validation and verification approach and tools.

Our design team has been spending some time the last few weeks at taking another look at our architectural practices and also at standardizing on a core Web application architecture that we can then drop into the heart of any web application and then extend/wrap it based on the needs of the particular web application we may be working on.

Performance is one of the key factors in determining best practices and for data intensive applications, the database and data access layer performance in particular can make or break an application’s responsiveness more than any other layer in the application.

As we do a lot of ASP.Net/ .Net development, one of our designers Sanguanchai who is based in our Bangkok office has been trying to break down the use of LINQ to SQL versus using the Entity Framework 4 (currently in beta 1) across key considerations – not just performance - for use in our core architecture.

You can take a look at his findings in the post he published yesterday - LINQ to SQL vs Entity Framework. No doubt there are other architects and developers out there trying to get a handle on the same question and we hope you find this useful.

On a slightly related note, if you are the podcast listening type, .Net Rocks had Dan Simmons, Dev Manager on the EF4 and LINQ to SQL Team as a guest on the latest show. It’s good listen and you get some background on Microsoft’s decisions – including why EF is at version 4 though it is actually the 2nd release. You also get a view into the decisions behind certain features in EF4 and I personally got a better understanding of how best to leverage certain features of EF4 that were opaque to me thus far.

Again, the direct link to Sanguanchai’s post:

http://www.osellus.com/blogs/2009/06/04/linq-to-sql-vs-entity-framework/

Enjoy!

I was reading a very interesting Reuters article about Honda’s new entry into the hybrid car market. It talks about how Honda has differentiated itself from Toyota (the previous market leader in this area) by incorporating an innovative software in its new hybrid model: Insight.

But pressure to innovate in the hybrid market won’t necessarily be all about the hardware. If hybrids and other cars embedded with electronic controls and more advanced computing power make their way into the mass market, opportunities could also arise for software. The Insight, for example, includes an “Eco-Assist” display on the dashboard that takes a cue from video games to help educate and encourage drivers to accelerate and decelerate for maximum fuel efficiency. As Wagner James Au explained in an Earth2Tech post about the feature earlier this year, “Like a role-playing game, the driver’s behavior is also tallied over time, and displayed symbolically — here, in the form of an ivy-ringed trophy achievement that a driver can gradually unlock with green-friendly driving. It’s sort of like Wii Fit, but for cars.” Depending on how much Honda decides to open that data, it could provide fodder for game developers, application platform providers and other industrious web entrepreneurs.

Software as a critical competitive advantage has already been a proven fact in the cell phone industry. Now it’s the auto industry’s turn. As software developers, we live in exciting times.

I have been reading Payman’s posts over the past week and the No new Methodology Please; Let’s focus on Tools post in particular stuck with me because it focuses on something I have been thinking about too. His post talks about the noise in the processes domain where there is a lot of focus on coming up with unnecessarily niche processes rather than focus on actually making processes useful by developing tools that make employing existing mature processes practical.

While Payman’s post focuses on processes, it struck me that even where tools go, often companies employ ALM tools with a narrow focus without fully grasping how they can be fully leveraged and what capabilities these tools pack. Over the last year, we have helped design and deploy solutions based on Microsoft Visual studio Team System (VSTS) for a few customers. One of the key needs that kept leaping out at us from all these engagements is that most customers wanted a way to implement requirements traceability – and they were not always aware that most ALM tools already have the capability to implement end-to-end Requirements Management – in particular traceability.

VSTS for example provides advanced linking capabilities that can be leveraged to allow tracking execution of a requirement from beginning to end (and backwards as well). A number of tools have come up around VSTS to satisfy other aspects of traceability such as the ability to track and link requirements within documents – document traceability.  Traceability has always been this much sought after goal – which many companies have been attempting to address in their tools for many years. Now, the tools exist but most of us are not aware how to leverage them or even that the capability exists.

Over the following few weeks, I am thinking of putting up a series of blog posts focusing on Traceability to think through for myself and others who are interested in Traceability – how to leverage tools – with particular focus on VSTS since I am most familiar with it. A recent solution we implemented for a customer involved integrating VSTS with their CRM solution so that the solution was now truly end-to-end : Right from the time the requirement came in from the customer. In a future post, I hope to outline the lessons we learnt from this particular implementation and also step back to examine the broader implications and advantages of implementing such integrations.

We are also discussing the possibility of perhaps even having a series of webinars that focus on various aspects of Traceability based on our experience over many projects over the years and  how we can leverage VSTS for the purpose.

Watch this space.

I have been keenly watching the progress on what started as a promising project in the Eclipse foundation. Unfortunately after more than two years of opportunity to shine I would have to give the EPF project a failing grade. I fear that when someone at IBM realizes the wasteful investment of resources in this project they will pull the plug on their participation at EPF. This will inevitably kill the project since IBM is the only committer on the tool aspect of EPF and the major committer and donor to the content aspect of EPF.

In my view there are several reasons for this failure and many lessons that I have learnt simply by following the travails of the project. These reasons are not the lack of aptitude and personal contribution of the EPF team but simply a miscalculated approach to building a vibrant self-organizing process community. Today I will highlight the mind boggling high transaction costs of submitting process content to be consumed and critiqued by a peer network. Even by their own admission, it takes “2-3 months” for the eclipse legal process to approve any process contributions to EPF. This is an anachronistic concept in today’s day and age. Communities are self-organizing. If what I contribute is useless to you, you will not use it. Period. Moreover having been tagged as someone who supplies useless content my user ratings will go down and I will have to significantly improve my game to get back my reputation. Communities give rise to wisdom-of-the-crowd, encourage meritocracy and do not tolerate back room machinations. As articulated by Chris Anderson in The Long Tail while discussing the tools needed to support a long tail, “This is the end of spoon-fed orthodoxy and infallible institutions, and the rise of messy mosaics of information that require - and reward – investigation”. Elsewhere in the book he discusses how “a once-monolithic industry structure where professionals produced and amateurs consumed is now a two-way marketplace, where anyone can be in any camp at any time. This is just a hint of the sort of profound change that the democratized tools of production and distribution can foster”. In the case of software process modeling, these tools take the form of easy to use process modeling and consumption tools. With that, rather than a paradigm of a handful of processes being donated by a few process experts, one could feasibly have hundreds of process variants created, put on wikis, blogged and rated! Now if we multiply the 2-3 months timeframe for these 100 processes the EPF legal staff will be kept busy for 20 years…just on this project! Later I will discuss some other pitfalls to avoid based on the lessons learned from following this project.

Yesterday Omid and I gave a joint Microsoft-Osellus webinar on Effective Software Development Governance with Adam Galant. Thanks to Rini Gahir and Adam Galant from Microsoft Canada for arranging this webinar.

It was a challenge to fit an explanation of the theoretical concepts of SD Governance with a tangible step-by-step approach to achieving it within the 45 minutes we had. In the end we were able to keep to our time and I hope the audience (about 36 attendees) left with an outline of a practical approach to establishing software development governance in an automated yet flexible manner.

Here is a link to a recording of the governance webinar.

You can use this link to register for the joint Microsoft-Osellus Webinar on software development governance. Omid is going to be talking about a lot of things that we have been able to achieve with the collaboration between the osellus process toolset/service offerings and the VSTS integrated ALM.

Here is a preview of what we will be talking about:

Effective governance aligns your software development investment with your business strategy. By ensuring continuous traceability of software development activities, governance leads to enhanced customer satisfaction and compliance with internal and external policies.

While most organizations use some measure of implicit governance, the full potential of a governance framework can be realized by taking three practical and explicit measures. The software development processes in the organization need to be selected and tailored in a manner that supports organization’s business goals. These processes need to be enforced in a streamlined manner. And, they need to be monitored to ensure corrective actions are taken in a timely manner.

In this webinar we outline a practical approach to establish software development governance in an automated yet flexible manner. We will discuss how the business objectives, standards and control mechanisms in the organization should be mapped to process elements such a roles, activities and work products that guide practitioners in a non-intrusive manner. Microsoft Visual Studio Team System (VSTS) and the IRIS process automation suite from Osellus will be used to demonstrate implementation of an effective governance framework in real-world scenarios.

David had this to say (quite disdainfully ) about my theorizing about getting data out of Jazz into MS Excel and MS Project:

Instead of a one-way data transfer from Jazz to Microsoft documents, I think a much more useful exercise would be to extend MS Office to make it Jazz aware. This would allow the Office applications to behave as Jazz clients much like Microsoft and others have done for VSTS. Without Jazz awareness, I feel that these other applications would be limited to reporting functions and be of little use to the project management community.

I see this more as an exercise in extending MS Office (or your application of choice) rather than extending Jazz and can understand why IBM would be looking for partners to provide such a solution for their customers.

Point is good. Point taken. I am doing precisely what Negin was complaining about - missing the project management perspective and looking at it from the developer’s perspective - but I did fool myself for a bit there that I had the answer signed, sealed and delivered.

The project manager would rather work directly from inside MS Excel, MS Project and interfaces she is familiar with - and it is far fetched to expect non-developer stake-holders to log into a Jazz client/web client and export into Project or report into Excel.

Also, Work Item information (for example) could be changing frequently and direct integration into MS Excel and MS Project provides an opportunity to refresh statuses and other work item information directly from inside a report that the project manager has open compared to having to export repeatedly from inside a Jazz client each time.

Not to fear. As with the Jazz-based solution I outlined yesterday, I have a similarly vague outline of how to do this - atleast for Excel:

As mentioned in my previous post, we already have a way to get the data off the Jazz server.

Microsoft has made it equally easy to develop applications that target the Office ecosystem through VSTO (Visual Studio Tools for Office - currently VSTO 2005). This means that we have application level add-in support for Office system applications like MS Excel.

Through VSTO, I can write a plug-in for Excel that puts a Jazz toolbar/ribbon in Excel and interfaces directly with the service interface of the Jazz Server to get the raw data required off Jazz. This VSTO plugin would then take care of the business logic of putting together and rendering different reports and charts based on pre-defined queries by leveraging the runtime support in MS Excel that VSTO allows.

Reverting to the developer perspective for a bit, VSTO is installed as an add-on on Visual Studio and provides very nice design-time support - just like Win-Forms controls. Makes it that much easier to develop these plug-ins.

Depending on the needs, in case we find that the Jazz service interface is too ‘raw’ for consumption, another option would be to have the Excel VSTO plug-in interact with the Jazz server through a custom Jazz service-plugin that takes care of sending the Excel plug-in well filtered and formatted data. This has some obvious advantages but in my opinion, has the decisive disadvantage of mandatorily requiring a Jazz server side plug-in without which the nice MS Excel plugin we developed will not work.

The part I am not sure about is how to implement a similar VSTO plug-in for MS Project. If I remember correctly, VSTO doesn’t offer support for MS Project development. I could be wrong.

Browsing the VSTO site is becoming a pain right now because I am on a train back to Toronto that has wifi access on board - but connectivity is sporadic at best. I’ll check up a little later and update this post. If it does have support, that’s that.
If not, a plugin can be written directly against the MS Project SDK to achieve our end.

[Update] There seem to be a couple of options for developing MS Project plugins:

• The version of VSTO that comes integrated with Visual Studio 2008 (currently in beta) has support for creating MS Project plug-ins.
• I guess you can create a COM add-in for MS Project using the MS Project SDK I mentioned above but that would be something to consider only if I couldn’t use VSTO.