We have been working on a lot of apps on mobile platforms, especially the iPhone and as I mentioned in my previous post, we keep coming back to the problem of data integrity – in the cases where there is a server component and data needs to be synced up with the server.

Apps that depend on data on a server have several options of accessing data on servers. There is the ubiquitous RSS – get and read feeds. This is great for reading data off a server but quickly loses relevance when the application has a more complex data model and the app needs to write data to the server. The problem of writing data to server is easily solved through web-services, REST services and the likes through which you can talk to a service that takes care of translating your data into the server’s database.

The problem however is keeping the data on the device in sync with the server component. This problem is prominent on mobile devices because while your app on the device is working and live, there is not guarantee of continuous sustained connectivity with the server. Network connectivity is unpredictable and unavailable based on where you are, there are other problems that may interrupt your application. For example on the iPhone, an incoming phone call may interrupt your app while the data is being synced with the server. Your app or your device may crash.

Given all these possible scenarios, how do we ensure that data on the device is always in sync with the data on the server and secondly how do we ensure the integrity of the data given the number of breakages and opportunities for data corruption given the very plausible scenarios I mentioned above.

Over the next few weeks, as we evolve and implement our solutions to this problem, I plan to write a series of posts that looks at this problem in depth and possible solutions and their feasibility. One of the obvious first things we looked at was what are apps out there currently doing to address this issue of data integrity? Answer? Surprisingly little. Most apps out there simply don’t allow the user to do anything that requires server connectivity if the apps detect that there is no connectivity. This is not a great solution, and is primitive – especially given what we have come to expect on the desktop. Solutions like Google gears and soon the offline support in HTML5 make what’s being done on the mobile platforms look primitive.

So, why are we looking at this problem now? Of course we are motivated by the fact that so many apps face this problem but there is no well defined solution and doesn’t look like anybody is looking at it. But perhaps more importantly, technologically the tools are in place at this moment in mobile platforms for us to come up with a solution. The tools and utensils are there, the ingredients are all laid out for us, nobody has come up with a recipe yet or even entered the kitchen.

The iPhone, the Android and soon the Blackberry with v5 of the Blackberry development platform have support for SQLite – a full fledged transactional light weight embeddable database engine on the device itself. The mobile platforms themselves have powerful development kits that allow us to come up with robust solutions in combination with SQLite (and other mobile database engines).  The tools are there.

Over the next few weeks, I am going to describe various possible solutions and think through their feasibility and suitability through my posts. I think that would be the best way to get peer feedback and ideas as we hardly know everything there is to know and your input would be invaluable.

Keep an eye on this space.

It’s been around 2 months since my last post! We have been very busy with some very interesting projects. This year has had a good start from the point of view of getting to work with some very new and cutting edge implementations.

My team just finished work on a custom design/modeling tool for Yardistry implemented in Papervision3D and Flex 3. I have not seen something so functional and complex implemented in 3D on the browser platform. I am pretty proud of the work the guys put into building this product both in terms of technology and usability. I really encourage you to go play with it and use it once it is out at http://designer.yardistrystructures.com/. The custom design tool should be going live there within the next few days. I’ll put out a post once it is live.

The other thing I have playing with a lot in the last few weeks is the iPhone development platform. The popularity of the platform just cannot be ignored and Payman, I and  a few others have been putting a lot of thought into leveraging this first really usable mobile platform to bring our ideas to life.

Every serious developer has looked at some mobile device or platform in the past few years and had a few ideas for it but been frustrated with the sheer unfriendliness of the platform both for the user and the developer. I think the iPhone has changed all that – it’s a very serious mobile computing platform.

Payman put out a CMMI Reference App very recently and you are going to see many more in the next few months. The CMMI app is free. I know a lot of you come from the process engineering background and might find it very useful. Please don’t forget to rate it and leave a comment! Here’s the direct link to it in the app store: Click here.

I have been working on a very exciting iPhone app myself. I hate to be a tease but I cannot talk about specifics. Lets just say that it leverages social networks like Facebook and twitter to bring something very practical and functional to a crucial experience in a person’s life. Sorry, but that’s about as specific as I can get. I wanted to bring it up so that I can talk about a couple of things I have been thinking about since I started working on the app.

If you are an iPhone user, you probably groaned when you saw the words “social network”, “Facebook” and “Twitter” in reference to an iPhone app. I understand. There were an estimated 65,000 third party apps in the App Store mid July and it sometimes look like all of them deal with social networks.

We understood this. However, there is a difference between focusing on social networks like most apps tend to do and what we are trying to do – which is leverage them with a lot of attention to providing users with the benefits of integration with their favourite social networking sites without becoming about the social networks.

What we are realizing is that in order to differentiate between our app and the other thousands of apps in the app store, we have to do two things well.

First, it is no longer enough to have an application that has no server component. If your app is dependent solely on third party sites like Facebook, Twitter etc, you are always bound tightly to their APIs, restrictions and performance. Of course you are getting a lot - like a ready made audience and functionality that you don’t want to re-invent. But remember that everybody else out there is also using the same APIs and they probably work exactly like your application.  You have to enhance that experience in some way. The best way to do that is to have your own custom server component that adds some richness to the experience. You can layer this server component on top of existing social network sites ideally or add it to the list in order to achieve differentiation. The server component may or may not have a Web UI but it probably will have web services that cater to your app specifically.

Secondly, the integration with social sites should be balanced so that a user’s audience or their friends on those sites are made aware of your activities through your app but the app itself should not become focused around the third party sites.

A good example that comes close to illustrating my points is the GAP StyleMixer iPhone app. The StyleMixer app allows you to create and style your own outfit on the iPhone. I like the way it handles sharing the outfits a user creates. It allows you to share your outfits with your Facebook friends but also to GAP’s own StyleMixer community.

By doing that it gives you your standard Facebook experience but also gives you a specialized GAP community experience that would not be possible inside Facebook. It also means that the focus is around the StyleMixer community. Support for  third party sites like Facebook, Twitter can be added, removed based on their popularity. Your app is not very dependant on them anymore and works better because of the specialization you have built using your own server component along with a hook into Facebook and twitter.

Perhaps some of what I say above flies in the face of current practices but if you are building a serious product, I would say that the ‘Cloud’ becomes a rich back-end for your app only if you have a say in it – and that means your own server component more often than not.

It is easy to think that a server component for a mobile application is over-kill. But thanks to the iPhone and potentially platforms like Android, the time has come where a mobile application  is serious business and can be the backbone of your enterprise. If companies are getting funded for their iPhone app ideas, then we better believe its time to consider building more than a client only experience.

Compare features for LINQ to SQL vs Entity Framework

Limitation for Entity Framework in VS2008 SP1

1. It will throw error in runtime if we use Contains statement. For example,
from p in context.Yard_Projects
where p_projectIDs.Contains(p.ProjectID)
select p
2. It will not generate method that map to store procedure that are not return as entity type. (We need to Entity Client to execute store procedure.) For example,
Void DeleteProjects(@CSVProjectIDs)
3. We need to map every column of a table in the storage schema. If some columns are not mapped, it will compile error.
4. If we delete a type from diagram, it’s difficult to put it back. We may manually update in .edmx file or delete old one and generate new model.
5. You don’t get a lot of control over the storage schema at all. What you mostly see is the client schema and the mapping to the storage schema.
6. If you want create methods that are map to store procedure that has the result from multiple tables, we need to manually modify section “SSDL” in the .edmx file.

Performance Test

I tried to test on LINQ to SQL and Entity Framework, I found the overall performance Entity Framework is better than LINQ to SQL. I separated test cases into 2 test groups.
1. Test select/create/update/delete for single table. You can see my test result in the “TestForSingleTable” sheet.
2. Test select/create/update/delete for single table with many associations. You can see my test result in te “TestForManyAssociation” sheet.
You can see the performance report in the link, PerformanceReport

Recommendation

I think we should use Entity Framework than LINQ to SQL because many reasons as follows.
1. It is more flexible to mapping entity model to database. You can map one class to multiple tables, using inheritance.
2. It supports many queries, you can use LINQ to Entities, Entity SQL, Object Services and Entity Client.
3. When the database schema is changed, you can synchronize entity model from latest database.
4. It has better overall performance when compare with LINQ to SQL.
5. It will be improved many features in the VS2010. You will create complex data type, generate database from entity model.
6. It supports many database vender other than SQL Server.

But disadvantage of using Entity Framework may be complexity and development time.

Refer to my performance report, if we use Entity Framework to implement in the data layer, we should use the following patterns.testperformancelinqtosql_entityframework
1. Use LINQ to Entity for simple queries.
2. Use Entity Client or Object to execute Store Procedure for complex queries (many joins, groups) or queries that perform delete many entities with contains many associations.
3. Use Object Service to insert, update, delete single/multiple entities.