The Mental Blog

Does Core Data Sync Quack?

Thu, 23 May 2013 08:01:00 -0400

After a spate of articles on the topic, it’s been quite a while since I posted anything about iCloud/Core Data sync. To be honest, I hit some serious stumbling blocks, and ended up giving up on it entirely.

I did end up shipping Mental Case for Mac with iCloud sync, but to do so I adopted a customized fork of the TICDS framework. Mental Case 2 for iOS is just around the corner, and I am using the same sync solution there. (If you are interested in going that route, I recommend first giving the main repository a try. They pulled back my iCloud changes recently.)

Since my original series of posts, many well-known developers have spoken out on the issues that still plague iCloud when used in combination with Core Data. Some have spoken out against the principle of building sync into Core Data directly, and others (eg Black Pixel, Bare Bones, Jumsoft) on their first-hand experiences.

Two years after the technology was first introduced, very few apps have shipped with iCloud/Core Data support. I know of only two — Time Butler and ToDoMovies — each of which has a relatively simple entity model, and very little data to transfer. Perhaps most telling is that the only Apple app that seems to use the technology is iTunes Movie Trailers, which presumably also has a very simple model and little data.

With WWDC just around the corner, we again start to ponder if this will be the year Apple gets iCloud right. But the more I dig into iCloud/Core Data sync, the more I have come to realize that even if it worked as designed, it still may be quite flawed as a solution. They may have gotten it wrong from the outset, and some design failures are probably not easily addressed. What follows is a list of what I think is fundamentally wrong with iCloud/Core Data’s design, leaving aside any of the practical failures that we have witnessed in the past.

iCloud/Core Data issues that may never be resolved:

1) Brent Simmons has already criticized the approach of coupling sync so tightly with Core Data. I don’t think the problem is that Core Data is involved — after all, what use would an object graph management framework be if it didn’t help you manage your objects, including synchronization — but I do agree that Core Data sync should not be so tightly coupled to a single storage mechanism, namely iCloud. By not allowing for extension to other storage facilities, Core Data sync is no option for many categories of app, including those not in an App Store, cross-platform apps, apps that require secure data storage, and apps with developers who simply don’t want to be locked into a single service. In short, sync in Core Data should be an interface that can work with many backends, not just iCloud.

2) Core Data sync currently offers no developer access to the process. It seems Apple expected to be able to make a completely generic sync algorithm that would work in every case, but that was not realistic. Five minutes thinking through the problems that can arise quickly tells you that it is not possible to come up with generic solutions for many cases. Developer intervention is needed in all but the most trivial examples. For example, there is no way for the developer to influence conflict resolution. If conflicts arise, Core Data either decides how to proceed itself, or halts syncing altogether. Some have endeavored to work within these constraints, using hacks to detect conflicts and recover (eg UbiquityStoreManager project, Tom Harrington), but these attempts just make it even more evident that the API of Core Data sync is fundamentally flawed.

3) Because the developer is not invited to the process, the first you get notified of sync changes is after they are resident in the persistent store. In other words, in many situations, your store will actually contain data in an invalid state. Apple recommends ‘cleaning up’ (eg deduping, applying validation constraints) when the merge notification is fired, but I would argue that this is too late. It is OK for a managed object context to be in an invalid state, but your on-disk store should never been invalid. The fact that Core Data sync requires that your store become invalid is a fundamental design flaw of the API.

4) Sync has been built into Core Data with minimal changes to the existing API. Basically, there are a few new notifications, and new metadata properties on the NSPersistentStore class. As a result, many aspects of sync are arguably too tightly coupled to existing classes. For example, if the user logs out of their iCloud account while an app is running, the developer must immediately tear down their Core Data stack, or expect a crash. Not only that, they must migrate data out of the persistent store to a new store, or move to a completely different (un-synced) store. It’s a big mess, and you are left wondering why. Why does the persistent store even need to know it is actively being synchronized via iCloud? Presumably, a better solution would be to have another class monitoring saves into the store, and handling that completely independently. The Core Data stack should not be so fragile, and should continue to function, with sane solutions to sync interruptions offered.

5) Related to 4). Changes to iCloud availability make a developer’s life very difficult, requiring reloading and migration of data stores. Apple produced code to demonstrate how to handle this a year after the technology was first introduced, but it is far from a satisfactory solution. It seems like they didn’t give much thought to store management until it was too late, and then had to resort to obscure workarounds to make the system work. Offering a new persistent store manager class that handles the most important cases would be more sensible and make the technology a lot easier for developers to adopt.

6) Take an app like ToDoMovies, turn on iCloud, and mess with it on two devices at the same time. It doesn’t take long before each device shows different data, and remains that way for all eternity, never ending up back in a fully-synced state. This points to pretty serious flaws in the algorithms being used behind the scenes. I would rather Core Data ensure exactly the same operations end up being applied to each data store, and then require developer intervention to manually massage a potentially invalid state before saving merges, than have data end up systematically out-of-sync.

Who knows, maybe Apple will get iCloud working well enough without addressing these issues to make it a success. But I doubt it. I fear Core Data sync in its current form is a lame duck.

Are Prices Too Low in the Mac App Store?

Thu, 15 Nov 2012 04:37:00 -0500

Since its inception, the Mac App Store has prompted discussion amongst Mac developers over pricing. Will there be a race to the bottom? Is it better to lower prices to reach a broader audience? Those types of questions have been — and continue to be — debated in the corridors of Apple developer conferences the World over.

At the recent Çingleton conference, ex-Apple employee Michael Jurewitz addressed the issue front on. I wasn’t actually at the conference, so I have only heard his arguments second hand, but it seems the gist was that most of the top grossing Mac apps have high prices, so Mac developers are encouraged to price their software higher.

It’s certainly an interesting analysis, but are the conclusions actually well established? Or is it another case of ‘damn lies and statistics’?

I decided to investigate this further in an impromptu study. I quickly realized that the results depend quite dramatically on exactly who you are. In particular, whether you are an established company like Apple or The Omni Group, or a small indie startup trying to establish a foothold.

My analysis was far from scientific, but it does point to some interesting conclusions. What I did was go through the Top 100 grossing apps, and I categorized each app as either

Developed by Apple or another big company (eg Adobe, EA)
Developed by a well-established company, or existing prior to the Mac App Store
A new app introduced after the Mac App Store launched

I discarded all apps in category 1, because how well Apple and Adobe do has little influence on the rest of us. I took the remaining results, for smaller companies, and tallied them up in the ranges 1-to-50, and 51-to-100, and took an average price for each range and product category. Here is what I found.

Product Category	Top Grossing Range	Number in Range	Average Price
Established apps	1-50	13	$68.83
Established apps	51-100	13	$55.91
New apps	1-50	6	$11.99
New apps	51-100	25	$18.31

The first thing you will notice is that the results confirm Michael Jurewitz’s analysis — there are more higher-priced apps in the top 50. Not only that, but the difference in average price for the ranges 1-50 and 51-100 is minimal, almost certainly within the statistical error of this very imperfect study. Conclusion: established companies have little to win by dropping their prices.

But that is perhaps asking the wrong question, at least for most of us. What indie developers want to know is: “If I start developing a new app, should I price it high or low?” The results for the newer apps shed some light on this. The average price of new apps in the top 50 is about a third lower than the average price of new apps in the 51-100 range. In other words, it seems you could mount an argument that new apps do better when they move to a lower price point.

I am not going to claim that what I have presented is scientific — for one thing, there is some subjectivity in how the apps are classified — but I think this should at least make us approach the question differently. Comparing indie developers to Apple, or even to well established developers like The Omni Group and Panic, doesn’t make sense, because the type of software they can produce is vastly different.

A quick statistical analysis does suggest that of the newer apps, the ones that gross the most are actually priced somewhat lower. Raising the price of your new release is more likely to do harm than good.

How to Choose Keywords for the App Store

Sun, 28 Oct 2012 06:00:00 -0400

When an app developer comes to release a new app, after months of development, they are inclined to relax too soon, and gloss over the submission process. But the choices you make when submitting an app can have a big impact on your visibility in the App Store. With almost a million apps vying for attention, details like screenshots, app description, title and keywords can make all the difference.

Nobody knows exactly what formula Apple uses to rank search results, but it is pretty clear it is a combination of text search, download stats, and ratings. The only aspect you have direct influence on is the text you enter. The title and keywords seem to be paramount. Terms in the full description are either not scanned at all, or weighted much more lightly.

My wife, Jennifer, just released her second app, Speeches. It’s an app for preparing and delivering slide-less presentations and pitches. Basically, speaker’s notes on your iPhone.

I helped her out with the submission, and came to some realizations about how to choose terms for the title and keywords. We established a small system for narrowing in on what would be most effective, and I want to outline what we came up with here.

The first thing you need to realize when introducing a new app is that you are already at an enormous disadvantage. There are probably existing apps in the App Store that are similar to your app, and they have a big lead in terms of ratings and downloads. Even if your app is much better, there is a good chance you will get completely buried at the bottom of the box, and nobody will find your app. It is essential that you do what you can to raise your visibility by careful choice of search terms.

The first step is just to write down all the terms you can think of that people might enter in a search for your app. Use a Thesaurus and Google searches for inspiration in order to expand on the terms you can think of immediately yourself. Don’t just come up with general terms, but seek out terms for very specific uses of your app. For example, obvious terms for Speeches include ‘public speaking’, ‘speech’, and ‘cue cards’, but terms like ‘wedding speech’ and ‘debating’ could be just as valuable.

The second stage is to test the words. How? It’s easy — just search for them in the App Store. Contrary to what you might expect, you are searching for terms that produce few hits. When your app enters the fray, it will likely be way down any list of results, so it is important to have short lists. Less than 10 results is ideal. Less than 20 is still acceptable, but more than that is probably a waste of valuable letters in your keyword field. People will simply not scroll through to your app if it is at number 36, and certainly not if it is at 413.

In the case of Speeches, we initially had terms like ‘keynote’ and ‘powerpoint’ on the list, but seeing these produced hundreds of hits, dismissed them as a lost cause. We focussed instead on terms, and combinations of terms that gave us the most chance of being visible to the user, even if the app was ranked at the back. At least then the app is visible, and it’s up to the icon and screenshots to convince a potential customer to look further.

With terms chosen, it’s just a question of filling the keyword field with what you think is most important. And don’t be afraid to include a byline in the App Store title, together with your App’s name. These terms are also weighted heavily. We went with “Speeches — Speaker’s notes for public speaking’. If the title gets truncated, at least the app’s name is still visible, and the byline both describes the app in one short sentence, and boosts those terms in the search results.

There’s no telling if Speeches will have any degree of success — the App Store is a hard mistress these days — but at least we have given it the best chance of being found by potential customers. And it’s an ongoing process too, of course; if you think of better keywords, you can rotate them in with any update.

Feeding the Main Queue with Grand Central Dispatch

Thu, 06 Sep 2012 08:57:00 -0400

So here’s the thing: you want to be a good citizen, and dispatch your expensive operations to a background queue, but it is impossible or difficult to do so because your code uses frameworks that must be accessed from the main thread. You could go to the effort of copying data into dictionaries and arrays, and passing those around, in order to avoid accessing the framework in the background, but it is a major headache.

Instead, you think, I could slice the problem up into small chunks of work, and run them on the main queue. That shouldn’t interrupt the user interface too much, which is why you need to avoid the main queue in the first place.

So you submit a few hundred blocks of work to the main queue, each of which does a very short calculation and completes. But to your horror, when you run the app, it beach balls regardless. The blocks you are submitting get queued up one after another, but then event handling blocks have to queue up and wait for all of your blocks to complete. Back to square one.

What you really need to do is feed one block to the main queue, and only feed the next one when the first is finished. That way, event handling can interleave with your work blocks. But how do you do that?

I’ve wrestled with this scenario a few times over the years, but I think I have finally found a good solution, and one which leaves the user interface flowing like soft butter. Not only that, it’s really straightforward.

The idea is to create a background serial queue, and enqueue blocks that submit your original blocks synchronously to the main queue.

-(void)enqueueBlock:(void (^)(void))block
{
    static dispatch_queue_t feederQueue = NULL;
    if ( !feederQueue ) feederQueue = 
        dispatch_queue_create("com.mentalfaculty.feederqueue", DISPATCH_QUEUE_SERIAL);
    dispatch_async(feederQueue, ^{
        dispatch_sync(dispatch_get_main_queue(), block);
    });
}

Then, you enqueue work like this

[self enqueueBlock:^{
    // Do some work on the main thread
    // Not too much!
    ...
}];

It’s simple, and best of all, it works a treat.

The next logical step would be a simple feeder class that can perform this sort of task for any queue you choose.

Babbleze and the 80/20 Rule

Thu, 05 Jul 2012 03:30:22 -0400

You won’t find a software developer in the World today who hasn’t heard of the 80/20 rule, but I’m guessing many don’t realize you can apply it to nearly anything. I first learnt this lesson from the The 4-Hour Work Week. If you can look past some of the hype, there are a lot of useful tips in the book.

One thing I am really bad at is applying 80/20 to my projects. For example, I’ve now spent many months trying to get my app to sync reliably via iCloud. That’s months of expensive, intense, and somewhat stressful engineering time. Luckily I don’t have to pay the bill, because it would be expensive if I did.

The idea of 80/20 as told in The 4-Hour Work Week is that you should pick the low hanging fruit when it comes to growing your business, the stuff that will give the most bang for your buck. For example, I could have spent the same time as I have spent on iCloud syncing developing a whole new iPhone app, and that may well have brought in more income than an incremental improvement to my existing app.

My wife, Jennifer, has just released her first app (or series of apps to be precise). Babbleze is an audio flashcard app for learning languages. Unlike my flagship app Mental Case, which is the powertool of flashcard apps, Babbleze is much simpler, but because it is focussed on one task, in my opinion it succeeds as a product.

Jennifer doesn’t have the years of programming experience that I have, and yet she found a category of apps that was within her range of skills, and then made a polished app in that category. To me, it’s a great example of 80/20. You don’t have to be the World’s best programmer to make a good app. You just have to think carefully about design, and make a polished product to solve a problem. In short, make the most of what you have.

Under the Sheets with iCloud and Core Data: Troubleshooting

Sat, 16 Jun 2012 15:34:07 -0400

I want to finish off this series with a post on troubleshooting, and I’m not going to sugar coat it — there’s lots of trouble to shoot.

You could find much of the material I covered earlier in the series elsewhere, but this post is based on the nitty-gritty, day-to-day issues you only encounter when you try to retrofit a shipping Core Data app with a cool new exhaust in the form of iCloud.

Today we’ll go through the stuff they didn’t teach you in Cocoa school. For many, it may well be the most useful post of the whole series.

Singletons

Many Core Data apps include what I will call singleton entities. Like singleton classes, these are entities for which there should only be one instance in the store. For example, many apps store certain settings or metadata in a single instance of an entity.

If you have entities like this, you will need to develop a strategy for ensuring uniqueness. If an instance is created on two different devices, you will end up with two instances on each device after iCloud merges changes.

There are two ways to approach this. You could simply check for extra instances after a merge, and use a deterministic system for removing them so that the same instances get removed on each device. For example, you could have a globally unique identifier or creation date attribute in the entity, such that you can sort instances and guarantee that the app is retaining the same instance each time.

Another approach is to create the singleton instances as soon as a new store is setup, and, on those devices seeded from iCloud, do not allow user interaction until the singletons have been merged and can be fetched. In this way, you ensure the singleton entities only get created once.

Side Effects in Accessors

When I first began to integrate iCloud into my Core Data apps, I held to a serious misconception which ended up wasting a considerable amount of time. That misconception was that my source code had no bearing on the Core Data transaction log import process. I thought that the import process was a private implementation detail; while that is largely true, you can in fact influence it, and sometimes that can have a negative impact.

The import takes place in a private context using a private persistent store coordinator, but it still makes use of your managed object classes, and uses key-value coding (KVC) to access properties. That means that if you have any side effects in your custom accessor methods, you could end up undermining the import.

In particular, I have found that creating or deleting objects in an accessor method leads to errors that prevent transaction log imports from completing. For example, take the following setter method.

-(void)setAppearsInSlideshows:(NSNumber *)yn
{
    [self willChangeValueForKey:@"appearsInSlideshows"];
    [self setPrimitiveValue:yn forKey:@"appearsInSlideshows"];
    [self updateFacetPermutations];
    [self didChangeValueForKey:@"appearsInSlideshows"];
}

The updateFacetPermutations method creates and deletes objects in order to keep the object graph in a valid state in normal app operation, but these side effects were causing the transaction log imports to fail. Creation and deletion of objects independent of the deltas in the change set is very likely to conflict with the objects being imported.

The solution to this problem is to ensure that your accessors remain as simple as possible. Use the vanilla methods provided by Core Data. If you need more advanced functionality in order to enforce validity in other parts of your code, either create a secondary accessor using a naming scheme not recognized by KVC, or introduce dependent properties.

To demonstrate these alternatives, consider again the setter shown above. The appearsInSlideshows property could be reverted to use the default accessor methods provided by Core Data. To ensure validity of the object graph when mutating in other parts of the source code, a second setter-like method, eg changeAppearsInSlideshowsTo:, could be introduced to take over the function of the original setter. The application code could use this method when changing the property, and the iCloud import would use the standard, unadulterated setter.

If you are developing for the Mac, and using Cocoa Bindings, you may be better off introducing a second, dependent property instead. Your interface can then bind to the dependent property, which includes any side effects to ensure object graph validity, while the iCloud import again adopts the standard accessors.

To demonstrate this option, here is the code I used in my own app for the appearsInSlideshows property.

+(NSSet *)keyPathsForValuesAffectingBindableAppearsInSlideshows
{
    return [NSSet setWithObject:@"appearsInSlideshows"];
}

-(void)setBindableAppearsInSlideshows:(NSNumber *)yn
{
    [self willChangeValueForKey:@"bindableAppearsInSlideshows"];
    self.appearsInSlideshows = yn;
    [self updateFacetPermutations];
    [self didChangeValueForKey:@"bindableAppearsInSlideshows"];
}

-(NSNumber *)bindableAppearsInSlideshows
{
    [self willAccessValueForKey:@"bindableAppearsInSlideshows"];
    id result = self.appearsInSlideshows;
    [self didAccessValueForKey:@"bindableAppearsInSlideshows"];
    return result;
}

The bindableAppearsInSlideshows property is used where previously the appearsInSlideshows property would have been used, including in bindings. The setBindableAppearsInSlideshows: method includes the side effects previously in the setAppearsInSlideshows: method, but these will be avoided during the iCloud import.

Validation Failures

Your app’s source can influence the import process in another way too. The validation rules of your entity model are applied during the import, as are validation checks in your managed object classes. If any fail, the import fails, and will never recover, leaving the app’s data in an inconsistent state across devices.

“But why would the validation ever fail?”, you might ask. Unfortunately, it turns out that it is much easier to cause a validation failure than you might expect. Any time you have two devices working simultaneously on the same data, the potential for conflict arises, and that could lead to a validation failure. (Note also that simply setting a merge policy is no guarantee that your managed object context will be left in a valid, savable state. This seems to be a common misconception.)

To demonstrate, consider the following simple entity model: Entity A has a to-one relationship to Entity B called b. Entity B has an inverse to-one relationship called a. Neither relationship is optional.

Assume we have two devices (1) and (2) that begin fully synced. Each has one object of class A, and one object of class B, and they are associated with one another. On device 1, we have objects A1 and B1, and on device 2 we have the same logical objects A1 and B1.

Now assume that simulateous changes are made on each device:

On device 1, we delete B1, insert B2, and associate A1 with B2. Then save changes.
On device 2, we also delete B1, insert B3, and associate A1 with B3. Then save changes.

Device 1 now attempts to import the transaction logs from device 2. B3 will be inserted, and A1 will be associated with B3. So far so good, but B2 is now left with relationship a equal to nil. This relationship is non-optional, so a validation error occurs.

Something similar will occur on device 2, because there are two B objects, and only one A object to associate with. There must thus always be a validation error, because one of the B objects must have its relationship set to nil.

Even worse, any future change will always leave an errant B object hanging around, and will thus fail validation. In effect, the user cannot fix the problem themselves by resetting the relationship. It is permanently broken.

This is more than a theoretical exercise. If you want to see for yourself, download the test app I introduced in previous posts, and carry out the following experiment.

Fill in your team identifier in the source code, and setup provisioning.
Build the test app on two Macs or virtual machines running Lion.
Add a new note on one machine, and start syncing.
Wait for the note to appear on the other machine.
Select the note on each machine, and press the ‘Change Schedule’ button on each.
Now press Save on each machine. Do this at almost the same time, so iCloud cannot sync in between.
Wait for the errors to appear in the console on each Mac.

The errors should look something like this (edited for readability):

2012-06-04 13:09:31.450 iCloudCoreDataTester[72072:239b] 
    -[_PFUbiquityRecordImportOperation main](435): CoreData: Ubiquity:  
    CoreData: Ubiquity: Error saving managed object context changes for transaction log: 
    <PFUbiquityTransactionLog: 0x7fd55ad06920>   
    transactionLogLocation: <PFUbiquityLocation: 0x7fd55ac33a00>: ...

Error: Error Domain=NSCocoaErrorDomain Code=1570 
    "Property/permutation/Entity/ChildSchedule is a required value." 
    UserInfo=0x7fd55ad318d0 
    {NSValidationErrorObject=<NSManagedObject: 0x7fd55ac3fa10> 
    (entity: ChildSchedule; id: 0x7fd55ac40a70 <x-coredata://01B641DC-E42D-4876-BB6E-7C9F33FEB704/ChildSchedule/p5> ; 
        data: {
            permutation = nil;
            title = "30AD7F4F-1109-44F9-996A-AAE9915150FD-955-0000051A8";
            }), 
        NSLocalizedDescription=Property/permutation/Entity/ChildSchedule is a required value., 
        NSValidationErrorKey=permutation}

and later

2012-06-04 13:09:31.451 iCloudCoreDataTester[72072:239b] 
    -[_PFUbiquityRecordsImporter operation:failedWithError:](824): 
    CoreData: Ubiquity:  Import operation encountered a corrupt log file, 
    Error Domain=NSCocoaErrorDomain Code=134302 
    "The operation couldn’t be completed. (Cocoa error 134302.)" UserInfo=0x7fd55ab035d0 
    {underlyingError=Error Domain=NSCocoaErrorDomain Code=1570 
    "Property/permutation/Entity/ChildSchedule is a required value." 
    UserInfo=0x7fd55ad318d0 {NSValidationErrorObject=<NSManagedObject: 0x7fd55ac3fa10>

These errors are telling you about exactly the scenario I described above: two new objects (ChildSchedule class) are vying for one vacancy in a to-one relationship. One of the objects is left with a nil value for the non-optional permutation relationship.

Living with Validation Failures

It is difficult to see how Apple could come up with a general solution to the problem of validation. In the scenario above, for example, Core Data could just delete the left over object, but that is a pretty extreme measure. The object may not even be new; it may be an existing object that was added to the relationship, so deleting it may have drastic consequences. The ultimate solution is probably that Apple needs to provide more hooks for the developer to resolve validation issues during import, but what do we do in the meantime?

One solution is to simply make all of the relationships in your model optional, or use weak relationships by storing the unique identifiers of related objects, rather than using an explicit relationship in the entity model. This will avoid the validation errors, but at a considerable cost to ease of coding. You will need to add a lot of code to take over the role that Core Data was playing, keeping your object graph in a valid state. At the very least, you will have to scan for stray objects after each merge, and delete or relocate them.

The solution I have chosen is to maintain the same Core Data entity model, with non-optional relationships, but to avoid any validation at all during the transaction log import. It is difficult for me to oversee all of the risks of this without intimate knowledge of the Core Data framework, so please be cautious if you decide to follow my lead. There may be unforeseen issues, but so far at least, it has been working OK for my app.

Here’s how it works: In order to disable validation during the transaction log import, I have overridden the NSManagedObject validateValue:forKey:error: method in a subclass used for all of my entities. The method will only validate if the managed object context class corresponds to custom subclass (MCManagedObjectContext) of my main context. If a standard NSManagedObjectContext instance is used, such as during the iCloud import, the method just returns YES whether the object is valid or not.

-(BOOL)validateValue:(__autoreleasing id *)value forKey:(NSString *)key error:(NSError *__autoreleasing *)error 
{
    if ( ![self.managedObjectContext isKindOfClass:[MCManagedObjectContext class]] ) {
        return YES;
    }
    else {
        return [super validateValue:value forKey:key error:error];
    }
}

The benefit of this is that validation errors are ignored during the import phase. They are effectively postponed, arising when you try to save your app’s primary context. At this point you have the means to address the errors and retry the save.

To give you some idea how this works, here is the save method used in my app.

-(void)save
{    
    [managedObjectContext performBlock:^{
        if ( managedObjectContext.hasChanges ) {
            NSUInteger attempts = 0;
            NSError *error = nil;
            while ( ![managedObjectContext save:&error] && ++attempts <= MCMaximumSaveAttempts ) {
                [self repairForSaveError:error];
            }

            if ( attempts > MCMaximumSaveAttempts ) {
                NSString *question = NSLocalizedString(@"A problem arose. Could not save changes.", @"Save fail");
                NSString *info = NSLocalizedString(@"You should quit as soon as possible, "
                    @"because continuing could cause other problems.", @"");
                [self runModalAlertWithMessage:question information:info];
            }
        }
    }];
}

If necessary, several attempts are made to save the context. If a save fails, a method is called to attempt to repair the validation problems.

-(void)repairForSaveError:(NSError *)error
{
    [managedObjectContext processPendingChanges];
    [managedObjectContext.undoManager disableUndoRegistration];

    if ( error.code != NSValidationMultipleErrorsError ) {
        id object = [error.userInfo objectForKey:@"NSValidationErrorObject"];
        [object repairForError:error];
    }
    else {
        NSArray *detailedErrors = [[error userInfo] objectForKey:NSDetailedErrorsKey];
        for ( NSError *error in detailedErrors ) {
            NSDictionary *detailedInfo = error.userInfo;
            id object = [detailedInfo objectForKey:@"NSValidationErrorObject"];
            [object repairForError:error];
        }
    }

    [managedObjectContext processPendingChanges];
    [managedObjectContext.undoManager enableUndoRegistration];
}

This method iterates over the validation errors and retrieves the object responsible for each. The violating object itself is then given the opportunity to address the error. The following code comes from the custom managed object class.

+(BOOL)deletesInvalidObjectsAfterFailedSave
{
    return NO;
}

-(void)repairForError:(NSError *)error
{
    if ( [self.class deletesInvalidObjectsAfterFailedSave] ) {
        [self.managedObjectContext deleteObject:self];
    }
}

The default repair is to do nothing. If an entity class overrides the deletesInvalidObjectsAfterFailedSave method and returns YES, any invalid object is simply deleted. Classes with more advanced requirements can override the repairForError: method to instigate repairs.

Debugging

One reason getting iCloud working in your Core Data app is so difficult is that debugging is torturously slow and frustrating. You build and run on two machines. Then you make a change on one machine, and wait…and wait…and… It’s even worse if iCloud decides to throttle syncing back, which is quite common. If that happens, you may be better off taking the rest of the day off.

To try to ease the pain a little, I’ve gathered some tips here to help you debug Core Data/iCloud apps. It won’t make it pleasant, but will hopefully save you a bit of time.

First, you need to decide what your debugging setup will be. You need two running OSes to test. You can’t run two apps in the same account, and share a single iCloud container. It would be nice, but you can’t. You also can’t really run in two separate user accounts and test using fast user switching. In my experience, you can get some strange locking issues with file coordinators. So you are left with two options: two different devices, or one device running a virtual machine.

I started testing between my iMac and MacBook Air, but after a few weeks elected to purchase VMware Fusion for my iMac, and do all my testing on that. I don’t regret that decision. It also makes it possible to test future OSes like Mountain Lion, so the virtual machine solution has many advantages.

Once you have a good testing setup, you will need a way to see what is going on. Core Data logs are extremely verbose when using iCloud, maybe too verbose. You will see a lot of messages; many of them will look extremely worrying, but are in fact completely harmless.

If you are debugging a specific problem, you may want to see every detail of the import process. In that case, you can make the console messages even more verbose by using the argument

-com.apple.coredata.ubiquity.logLevel 3

when launching your app. You set this in the Run section of your scheme, but be warned — you will get an awful lot of output.

Glossary of Innoculous Errors

Because the console output of Core Data can be scary to the newcomer, I’ve gathered together a few of the innocuous error messages to help you discern the forest from the trees.

The following occur all the time, and just mean the import couldn’t proceed due to missing files and the like. Core Data will try again in a minute or so, so just ignore these and wait.

2012-06-04 13:09:01.295 iCloudCoreDataTester[72072:6f23] +[PFUbiquityTransactionLog loadPlistAtLocation:withError:](378): 
CoreData: Ubiquity:  Encountered an error trying to open the log file at the location: <PFUbiquityLocation: 0x7fd55ac373a0>: ...
    Error: Error Domain=NSCocoaErrorDomain Code=256 "The file “67BFAE7B-9CE3-432A-AD1F-6EA23C648017.1.cdt” couldn’t be opened." 
    UserInfo=0x7fd55ac3b210 {NSURL=file://localhost...cdt, NSDescription=The item failed to download.}

2012-06-04 13:09:01.298 iCloudCoreDataTester[72072:6f23] -[PFUbiquityTransactionLog loadComparisonMetadataWithError:](244): 
CoreData: Ubiquity:  Error encountered while trying to load the comparison metadata for transaction 
log: <PFUbiquityTransactionLog: 0x7fd558e68a60>

This one seems to arise when Core Data is trying to resolve certain conflicts. It seems to be harmless too.

2012-06-06 17:13:05.913 Mental Case[854:ae53] CoreData: warning: An NSManagedObjectContext delegate overrode fault handling 
behavior to silently delete the object with ID '0x7fdae4dcd3c0 <x-coredata://367CA83F-D0FE-4E19-A52F-873A45EC954C/MCFacetPermutation/p5166>' 
and substitute nil/0 for all property values instead of throwing.

And lastly, the next one is actually a sign that an import was successful. At least, it seems to appear just after a successful import. I actually use it as a notification that the import has gone through.

2012-06-06 16:20:38.711 Mental Case[3631:750f] -[_PFUbiquityStack initWithLocalPeerID:andUbiquityRootLocation:](83): 
CoreData: Ubiquity:  Error encountered while trying to connect to the metadata store: Error Domain=NSCocoaErrorDomain Code=512 
"The file couldn’t be saved." UserInfo=0x7f9b7b41d8f0 {}

Where to Now?

That’s it for this series. What can we conclude? Unfortunately, the most apt conclusion is probably that iCloud syncing of Core Data is not really ready for prime time, at least not for any app with a complex data model. If you have a simple model, and patience, it is doable, even if very few have achieved a shipping app at this point.

Is the future brighter? We have Mountain Lion and iOS 6 just around the corner, and while Apple seems to have addressed some concerns, I still have plenty of reservations. Maybe I’m wrong, I even hope that I am, but only time will tell.

The promise of iCloud is great, but syncing is a very difficult problem, and the complexity of the incremental updates that Core Data introduces doesn’t make it any easier. I think Apple will eventually crack it, but it could be a bumpy ride in the meantime. I hope this series has at least taken some of the rough edges off.

Go forth and sync!

Drew McCormack

Under the Sheets with iCloud and Core Data: Sentinels

Fri, 08 Jun 2012 14:53:14 -0400

In the previous posts I alluded to some exceptional circumstances that can arise and need to be handled in a production app. In particular,

If an app is running, and the user disables iCloud, or deletes the iCloud container on any device, the app will crash.
If an app has been syncing at some point in the past, and undergoes a stoppage — a period of time where it does not sync — it can’t pick up again from where it left off. When enabling syncing, there needs to be a way to determine if a device ever contributed to the transaction logs present in the iCloud container.

One solution to these problems is to add a file or files to the iCloud container to track which devices have contributed to the transaction logs. When a device starts syncing, an entry is made in the container. When the device stops syncing, the device entry is left in place as a record that it contributed to the iCloud logs.

If there is an attempt to enable syncing on a device which was syncing in the past, it will be clear from the device entries that a stoppage has arisen, and the existing transaction logs can be removed. And if the container is deleted altogether, the device records will also get deleted. The app can detect the missing files, tear down its Core Data stack, and, for example, setup again with a non-ubiquitous store, avoiding a crash and/or data corruption.

I will refer to the files used to track syncing devices as ‘sentinels’. This term was introduced by Daniel Pasco in a presentation at NSConference 2012. Daniel used a very similar technique to what I will describe here, and has posted his code on GitHub.

The Sentinel

Daniel’s solution involves adding a file for each device that contributes to the transaction logs. My solution, which was developed independently, is similar in nature, but uses just a single properly list file holding an array of device identifiers. Each time a device starts syncing, it is added to this plist file.

The plist can be used to check whether a device for which syncing is about to be enabled has contributed to the transaction logs in iCloud at some point in the past. And if the plist file is deleted, or a device with syncing enabled suddenly no longer appears in the list of identifiers, it can be concluded that a reset has occurred, and evasive action can be taken.

The Sentinel Class

I have developed a class that creates and tracks a sentinel file. To get the source code, visit the test project introduced last time on GitHub, or — if you cloned the source last time — pull down the latest changes using git.

The sentinel class is called MCCloudResetSentinel. To create an instance, you use the initWithCloudStorageURL:cloudSyncEnabled: initializer. The first argument should be a URL for a directory in the app’s iCloud container. The class will create the sentinel plist file in this directory.

The second argument indicates whether syncing is currently enabled. This relates to the in-app sync setting, and not to whether iCloud is enabled globally for the device. The MCCloudResetSentinel class always assumes iCloud is globally enabled; if that is not the case, the class serves no useful purpose, as it has no access to the iCloud container.

The sentinel class is immutable. Once you initialize it, either for the syncing or non-syncing state, you can’t change it. If the syncing state does need to change, simply release the existing sentinel instance, and make a new one.

The MCCloudResetSentinel has a delegate, and will invoke the delegate method cloudResetSentinelDidDetectReset: if the current device identifier disappears from the list of syncing devices while syncing is enabled. It will only invoke this method once; a new sentinel instance should be created after the reset has been addressed.

The class also posts a MCCloudResetSentinelDidDetectResetNotification notification when a reset occurs, so objects other than the delegate can also take appropriate action, such as release references to invalidated managed objects.

When a device starts syncing, the sentinel object can be requested to add it to the list of devices using the method updateDevicesList:. Because this method has to ensure it has the very latest version of the devices property list, which may involve downloading from iCloud servers, the method is asynchronous, and takes a completion handler block as sole argument.

The last piece of major functionality in the MCCloudResetSentinel class is the ability to check if the current device is in the property list, ie, has at some point contributed to the iCloud transaction logs. The asynchronous checkCurrentDeviceRegistration: method is used for this purpose. It invokes a completion handler, passing YES if the device is listed.

Using MCCloudResetSentinel

The AppDelegate class includes a number of changes to make use of the sentinel class, and the UI of the test app has also been improved somewhat. For example, the interface now shows icons to indicate whether iCloud syncing is active, and whether the Core Data stack is currently setup.

The sentinel is used in a few places. First, when the app launches, a check is done to see whether a cloud reset has occurred while the app was not running. This occurs before the Core Data stack is initialized.

[self checkIfCloudDataHasBeenReset:^(BOOL hasBeenReset) {
    if ( hasBeenReset ) [self disableCloudAfterResetAndWarnUser];
    [self setupCoreDataStack:self];
}];

The checkIfCloudDataHasBeenReset: method uses a sentinel to test for a reset.

-(void)checkIfCloudDataHasBeenReset:(void (^)(BOOL hasBeenReset))completionBlock
{
    dispatch_queue_t completionQueue = dispatch_get_current_queue();
    dispatch_retain(completionQueue);

    BOOL usingCloudStorage = 
        [[NSUserDefaults standardUserDefaults] boolForKey:MCUsingCloudStorageDefault];
    if ( usingCloudStorage && !self.cloudStoreURL ) {
        dispatch_async(completionQueue, ^{
            completionBlock(YES);
            dispatch_release(completionQueue);
        });
        return;
    }

    // Use a temporary sentinel to determine if a reset of cloud data has occurred
    MCCloudResetSentinel *tempSentinel = 
        [[MCCloudResetSentinel alloc] initWithCloudStorageURL:self.cloudStoreURL 
                                             cloudSyncEnabled:usingCloudStorage];
    if ( usingCloudStorage ) {
        [tempSentinel checkCurrentDeviceRegistration:^(BOOL deviceIsPresent) {
            dispatch_async(completionQueue, ^{
                completionBlock(!deviceIsPresent);
                dispatch_release(completionQueue);
            });
        }];
    }
    else {
        dispatch_async(completionQueue, ^{
            completionBlock(NO);
            dispatch_release(completionQueue);
        });
    }
}

This code is asynchronous and makes heavy use of Grand Central Dispatch (GCD), but the idea is fairly straightforward: if cloud syncing is enabled, which is determined by a user default, a temporary sentinel is created to check if the current device is in the device list. The result is passed back to the completion block.

Whenever the Core Data stack is setup, a sentinel is created to update the devices list, and then monitor it continuously for iCloud container resets.

__weak AppDelegate *weakSelf = self;
[self addStoreToPersistentStoreCoordinator:^(BOOL success, NSError *error) {
    if ( !success ) {
        [weakSelf tearDownCoreDataStack:weakSelf];
        [[NSApplication sharedApplication] presentError:error];
    }
    else {
        [weakSelf makeManagedObjectContext];

        // Setup a sentinel
        BOOL usingCloudStorage = 
            [[NSUserDefaults standardUserDefaults] boolForKey:MCUsingCloudStorageDefault];
        if ( usingCloudStorage ) {
            weakSelf->sentinel = 
                [[MCCloudResetSentinel alloc] initWithCloudStorageURL:weakSelf.cloudStoreURL 
                                                     cloudSyncEnabled:usingCloudStorage];
            weakSelf->sentinel.delegate = self;
            [weakSelf->sentinel updateDevicesList:NULL];
        }

The other place a sentinel is used is when the user enables syncing. At this point, the sentinel is used to check if the device ever contributed transaction logs, and act accordingly.

// Use sentinel to determine if device was previously syncing.
// In that case, the only option is to replace the whole cloud container.
// If the device never synced before, the user can choose to keep the 
// local or the cloud data.
MCCloudResetSentinel *tempSentinel = 
    [[MCCloudResetSentinel alloc] initWithCloudStorageURL:self.cloudStoreURL 
                                         cloudSyncEnabled:NO];
[tempSentinel stopMonitoringDevicesList];
[tempSentinel checkCurrentDeviceRegistration:^(BOOL deviceIsPresent) {
    if ( deviceIsPresent ) {
        // Only choice is to move data to the cloud, replacing the existing cloud data.
        // In a production app, you should warn the user, and give them a chance
        // to back out.
        [self migrateStoreToCloud];
    }
    else {
        // Can keep either the cloud data, or the local data at this point
        // In a production app, you could ask the user what they want to keep.
        // Here we will just see if there is cloud data present, and if there is,
        // use that. If there is no cloud data, we'll keep the local data.
        BOOL migrateDataFromCloud = 
            [[NSFileManager defaultManager] fileExistsAtPath:self.cloudStoreURL.path];
        if ( migrateDataFromCloud ) {
            // Already cloud data present, so replace local data with it
            [self removeLocalFiles:self];
        }
        else {
            // No cloud data, so migrate local data to the cloud
            [self migrateStoreToCloud];
        }
    }
    [[NSUserDefaults standardUserDefaults] setBool:YES 
                                            forKey:MCUsingCloudStorageDefault];
    [self setupCoreDataStack:self];
}];

The comments in the code describe the various branches that can be taken. In short, if the device is in the list already, the only way to get a consistent set of data is to replace the iCloud container with data from the local store. If the device is not in the list, the option arises to either replace the iCloud data, or replace the local store.

Metadata — Faster than Some Speeding Data

I will use the rest of this post to go through the code of the sentinel class itself, because it is useful to understanding iCloud and how you interact with it.

One aspect of iCloud’s implementation that is useful to grasp is that it pushes metadata between devices and the cloud much faster than it moves file data itself. An app on one device can learn of a change on another device long before the file change itself has propagated over.

You can use the NSMetadataQuery class to monitor changes in file metadata, and Apple have extended the API to support ubiquitous data in the iCloud container. This is the same class you use to do Spotlight searches, and also underpins Time Machine backups.

You can do asynchronous searches with NSMetadataQuery, but also be notified of file metadata changes as they occur. The latter is where our interest lies here. The MCCloudResetSentinel class sets up a query in its initializer that matches the filename of the devices property list, and then listens for NSMetadataQueryDidUpdateNotification notifications from the query.

// Listen for changes in the metadata of the devices list
devicesListMetadataQuery = [[NSMetadataQuery alloc] init];
devicesListMetadataQuery.searchScopes = 
    [NSArray arrayWithObject:NSMetadataQueryUbiquitousDataScope];
devicesListMetadataQuery.predicate = 
    [NSPredicate predicateWithFormat:@"%K like %@", NSMetadataItemFSNameKey, MCSyncingDevicesListFilename];
[[NSNotificationCenter defaultCenter] addObserver:self selector:@selector(devicesListDidUpdate:) 
    name:NSMetadataQueryDidUpdateNotification object:devicesListMetadataQuery];
if ( ![devicesListMetadataQuery startQuery] ) NSLog(@"Failed to start devices list NSMetadataQuery");

The predicate of the query is used to restrict notifications to files named the same as the devices property list file, and the scope of the query — NSMetadataQueryUbiquitousDataScope — means only files in the iCloud outside of the Documents folder will be monitored. (If you want to monitor files inside the iCloud Documents folder, use NSMetadataQueryUbiquitousDocumentsScope.)

Downloading Files

Because metadata moves faster than file data, there may be times where your app knows about a change before it has happened, and needs to get hold of the new data as quickly as possible, rather than just waiting for iCloud to do its thing. This is the case, for example, with the devices list: if a reset has occurred, we want to know about it as soon as possible, so that we can take evasive action.

Luckily, Apple have built a few features into NSURL and NSFileManager to help with this. First, you can query the download state of a particular file using NSURL’s getResourceValue:forKey:error:, passing in NSURLUbiquitousItemIsDownloadedKey as the key. The value returned by reference contains an NSNumber holding a BOOL to indicate if the file is downloaded, or still has changes in the cloud.

If a file is not downloaded, you can force it to download using the startDownloadingUbiquitousItemAtURL:error: method. While it is downloading, the resource key NSURLUbiquitousItemIsDownloadingKey can be used with getResourceValue:forKey:error: to check if it is still downloading.

A lot of the time, you only care to ensure that you have the latest version of a file, and don’t need to know the details of how it is retrieved. For that reason, I added a category method to NSURL that makes sure the latest version of an iCloud file is present on the local disk.

-(void)syncWithCloud:(void (^)(BOOL success, NSError *error))completionBlock
{
    NSError *error;
    NSNumber *downloaded;
    BOOL success = [self getResourceValue:&downloaded forKey:NSURLUbiquitousItemIsDownloadedKey error:&error];
    if ( !success ) {
        // Resource doesn't exist
        completionBlock(YES, nil);
        return;
    }

    if ( !downloaded.boolValue ) {
        NSNumber *downloading;
        BOOL success = [self getResourceValue:&downloading forKey:NSURLUbiquitousItemIsDownloadingKey error:&error];
        if ( !success ) {
            completionBlock(NO, error);
            return;
        }

        if ( !downloading.boolValue ) {
            BOOL success = [[NSFileManager defaultManager] startDownloadingUbiquitousItemAtURL:self error:&error];
            if ( !success ) {
                completionBlock(NO, error);
                return;
            }
        }

        // Download not complete. Schedule another check. 
        double delayInSeconds = 0.1;
        dispatch_time_t popTime = dispatch_time(DISPATCH_TIME_NOW, delayInSeconds * NSEC_PER_SEC);
        dispatch_queue_t queue = dispatch_get_current_queue();
        dispatch_retain(queue);
        dispatch_after(popTime, queue, ^{
            [self syncWithCloud:[completionBlock copy]];
            dispatch_release(queue);
        });
    }
    else {
        completionBlock(YES, nil);
    }
}

This uses the methods discussed above to download the latest version, and waits until it is finished, before calling back to a completion block.

File Presenters

Unfortunately, I have found that sometimes the metadata query notification does not fire when the device list file is deleted. I suspect this may have something to do with the file deletion taking place at the same time as the metadata update arriving, but I’m not certain.

To make doubly sure that the sentinel class observes all changes to the device list file, I have made it conform to the NSFilePresenter protocol.

File presenters are the flip side of file coordinators, which we met last time. File coordinators provide a locking mechanism for safely manipulating files, and notify any file presenters when things change.

So in order to monitor changes to a file, you can register a file presenter, which is an object conforming to the NSFilePresenter protocol. The protocol requires the following methods.

-(NSURL *)presentedItemURL
{
    return self.syncedDevicesListURL;
}

-(NSOperationQueue *)presentedItemOperationQueue
{
    return filePresenterQueue;
}

The presentedItemURL returns the URL of the presented file, and presentedItemOperationQueue should provide the queue that is used for callbacks when the file changes.

The methods for observing changes are as follows:

-(void)presentedItemDidChange
{
    [[NSOperationQueue mainQueue] addOperationWithBlock:^{
        [self devicesListDidUpdate:nil];
    }];
}

-(void)accommodatePresentedItemDeletionWithCompletionHandler:(void (^)(NSError *errorOrNil))completionHandler
{
    [[NSOperationQueue mainQueue] addOperationWithBlock:^{
        [self devicesListDidUpdate:nil];
    }];
    completionHandler(nil);
}

The first is for a change to the file, and the second is for a deletion. In both cases, we queue up a device list check on the main queue.

The only other aspect of NSFilePresenter that should be considered is how it starts and stops monitoring. To begin monitoring the file, you actually need to use the NSFileCoordinator class.

    // Register as file presenter
    [NSFileCoordinator addFilePresenter:self];

And to stop monitoring, you remove the presenter.

-(void)stopMonitoringDevicesList
{
    [NSFileCoordinator removeFilePresenter:self];

Last Throw…

That’s it for sentinels. Next time I will finish off the series by discussing troubleshooting and debugging.

Under the Sheets with iCloud and Core Data: Working Code

Tue, 29 May 2012 15:19:49 -0400

Having covered Core Data syncing via iCloud from a high level, in this post I want to introduce a simple test app, which will be extended over the coming weeks. The app should eventually contain most of the code snippets you will need to get your own app up and running with iCloud.

Installing the Test App

The test app is available on github. You can download the source code as a zip archive, but it is probably best to use git to clone the project to your Mac, so that you can pull future changes as we go.

Once you have the source code, there are a few modifications you will need to make if you want to sync via iCloud. First, go into the AppDelegate.m file, and enter your Team ID at the top.

static NSString * const TeamIdentifier = @"XXXXXXXXXX";

Simply replace the placeholder string with your Team ID.

Next, you will need to go to the Developer Certificate Utility from the Mac Dev Center, and create a provisioning profile, as described in the first post. It is probably easiest to create the profile for a wildcard app. To do this, just use an asterisk (*) as the app bundle identifier. Once you download the provisioning profile, install it on your Mac by double clicking the file in Finder. Also add it to the Provisioning Profiles section under the Devices tab in the Xcode Organizer. Lastly, go into the iCloudCoreDataTester Target’s Build Settings and set the Code Signing Identity to use the new profile.

Introduction to iCloudCoreDataTester

The app is structurally very simple: all the interesting code is in the AppDelegate.m file. The Core Data entity model is based on parts of the model from my app, Mental Case, but is otherwise nonsense. Best not to try to understand what any of it means; just note the types of relationships involved, the deletion rules, and other aspects that may be of interest. The objective in designing the entity model was to mimic the sort of complex relationships that you might find in a production app.

The UI of the app is very straightforward. The list on the left shows a list of Note objects, which you can add and remove using the buttons underneath. If you select a note, a child object appears in the list on the right. This ‘schedule’ object can be switched using the Change Schedule button, which deletes the existing instance, and inserts a new one.

The model includes various hidden entities, which don’t appear in the UI, and are mainly there to setup different types of relationships, to thoroughly test iCloud.

At this point, the app does not handle stoppages. It is more a developer test apparatus than a user-friendly, end-user app. There are buttons to manually setup and tear down the Core Data stack, delete the local store, delete the cloud container, save the managed object context, and turn on iCloud syncing. (An entry in NSUserDefaults is used to persist whether iCloud syncing is enabled or not.)

The Core Data Stack

The AppDelegate begins with a number of simple methods for retrieving URLs for the local store and cloud container. Local data is located in the user’s Application Support folder

-(NSURL *)localStoreURL
{
    return [self.applicationFilesDirectory URLByAppendingPathComponent:@"iCloudCoreDataTester.storedata"];
}

and the cloud container is setup in a subpath (MainStore) of the main container.

-(NSURL *)cloudStoreURL
{
    NSString *bundleId = [[NSBundle mainBundle] bundleIdentifier];
    NSString *ubiquityId = [NSString stringWithFormat:@"%@.%@", TeamIdentifier, bundleId];
    NSURL *ubiquitousURL = [[NSFileManager defaultManager] URLForUbiquityContainerIdentifier:ubiquityId];
    NSURL *storeURL = [ubiquitousURL URLByAppendingPathComponent:@"MainStore"];
    return storeURL;
}

Each of the buttons in the UI is connected to a basic action. You will probably want similar methods in your own app, so that it is trivial to setup and teardown the Core Data stack, remove the cloud container, or perform other fundamental tasks at will.

-(IBAction)tearDownCoreDataStack:(id)sender
{
    if ( !stackIsSetup ) return;

    NSError *error;
    if ( ![self.managedObjectContext save:&error] ) {
        [NSApp presentError:error];
    }
    stackIsSetup = NO;

    [self.managedObjectContext reset];
    self.managedObjectContext = nil;
    self.managedObjectModel = nil;
    self.persistentStoreCoordinator = nil;
}

-(IBAction)setupCoreDataStack:(id)sender
{
    if ( stackIsSetup ) return;
    stackIsSetup = YES;
    [self willChangeValueForKey:@"managedObjectContext"];
    [self didChangeValueForKey:@"managedObjectContext"];
}

The iCloudCoreDataTester app uses Cocoa Bindings in the user interface, so setting up the Core Data stack just involves firing KVO notifications, which cause the table views to reload, and thereby lazy creation of the Core Data stack objects (eg NSPersistentStoreCoordinator, NSManagedObjectContext, etc).

Creation of the Core Data stack objects is fairly standard, and follows the prescription given in the first post. The persistent store coordinator adds the local store with ubiquity options if iCloud syncing is enabled.

// Use cloud storage if iCloud is enabled, and the user default is set to YES.
NSURL *storeURL = self.cloudStoreURL;
BOOL usingCloudStorage = [[NSUserDefaults standardUserDefaults] boolForKey:UsingCloudStorageDefault];
usingCloudStorage &= storeURL != nil;

// Basic options
NSMutableDictionary *options = [NSMutableDictionary dictionaryWithObjectsAndKeys:
    (id)kCFBooleanTrue, NSMigratePersistentStoresAutomaticallyOption, 
    (id)kCFBooleanTrue, NSInferMappingModelAutomaticallyOption, 
    nil];

// iCloud options
if ( usingCloudStorage ) {
    [options addEntriesFromDictionary:[NSDictionary dictionaryWithObjectsAndKeys:
         MCCloudMainStoreFileName, NSPersistentStoreUbiquitousContentNameKey,
         storeURL, NSPersistentStoreUbiquitousContentURLKey, 
         nil]];
}

Merging Changes

When the persistent store coordinator is created, the AppDelegate registers to receive notifications of iCloud merges.

// Register as observer for iCloud merge notifications
[[NSNotificationCenter defaultCenter] removeObserver:self];
[[NSNotificationCenter defaultCenter] addObserver:self selector:@selector(persistentStoreCoordinatorDidMergeCloudChanges:) 
    name:NSPersistentStoreDidImportUbiquitousContentChangesNotification object:coordinator];

When the notification is sent, it is passed directly to the mergeChangesFromContextDidSaveNotificaton: method of the NSManagedObjectContext.

-(void)persistentStoreCoordinatorDidMergeCloudChanges:(NSNotification *)notification
{
    [self.managedObjectContext performBlock:^{        
        [self.managedObjectContext mergeChangesFromContextDidSaveNotification:notification]; 

        NSError *error;
        if ( ![self.managedObjectContext save:&error] ) {
            [NSApp presentError:error];
        }
    }];
}

File Coordination

When Apple developed iCloud, they came across issues that don’t affect an isolated desktop app. One of those issues was that, with iCloud in the mix, files often need to be accessed from multiple processes at once. Apple needed to develop a robust locking and notification mechanism for file access, and that became the file coordination and presentation classes.

Whenever you need to perform any action at all on a file (or folder) in the iCloud container, you should use an NSFileCoordinator, to make sure you play nicely with the iCloud processes that are monitoring and modifying the same files. In iCloudCoreDataTester, an NSFileCoordinator is used when removing the iCloud container.

-(IBAction)removeCloudFiles:(id)sender
{
    [self tearDownCoreDataStack:self];

    [[NSUserDefaults standardUserDefaults] setBool:NO forKey:UsingCloudStorageDefault];

    NSFileCoordinator* coordinator = [[NSFileCoordinator alloc] initWithFilePresenter:nil];
    NSURL *storeURL = self.cloudStoreURL;
    if ( !storeURL ) return;
    [coordinator coordinateWritingItemAtURL:storeURL options:NSFileCoordinatorWritingForDeleting error:NULL byAccessor:^(NSURL *newURL) {
        [[NSFileManager defaultManager] removeItemAtURL:newURL error:NULL];
    }];
}

The method coordinateWritingItemAtURL:options:error:byAccessor: is responsible for retreiving a lock on the container’s folder URL, in order to perform our file operation, in this case a deletion (NSFileCoordinatorWritingForDeleting). In doing so, it informs any NSFilePresenter objects that are monitoring the container what is about to happen. In this way, iCloud will notice that the data has been deleted.

The actual removal is carried out using the default NSFileManager in a block passed as last argument to the method. You may think that this happens asynchrously, as is usually the case with block arguments, but the NSFileCoordinator methods are synchronous — the block has been executed, and the container deleted, when the method returns.

Note also that you should operate on the URL passed into the block, rather than the original storeURL. This is a good habit to get into, because with some file operations, it is possible the file’s URL may have been modified by previous operations.

Migration

Migrating an existing local store to iCloud is the task of the migrateStoreToCloud method. It begins by moving the local store to a new URL.

-(void)migrateStoreToCloud
{
    NSError *error;
    NSURL *storeURL = self.localStoreURL;
    NSURL *oldStoreURL = [[self applicationFilesDirectory] URLByAppendingPathComponent:@"OldStore"];
    NSFileManager *fileManager = [NSFileManager defaultManager];

    // If there is no local store, no need to migrate
    if ( ![fileManager fileExistsAtPath:storeURL.path] ) return;

    // Remove any existing old store file left over from a previous migration
    [fileManager removeItemAtURL:oldStoreURL error:NULL];

    // Move existing local store aside
    if ( ![fileManager moveItemAtURL:storeURL toURL:oldStoreURL error:&error] ) {
        [[NSApplication sharedApplication] presentError:error];
        return;
    }

Store options are then generated for the existing non-ubiquitous store, and the new ubiquitious store.

    // Options for new cloud store
    NSDictionary *localOnlyOptions = [NSDictionary dictionaryWithObjectsAndKeys:
        (id)kCFBooleanTrue, NSMigratePersistentStoresAutomaticallyOption, 
        (id)kCFBooleanTrue, NSInferMappingModelAutomaticallyOption, 
        nil];
    NSDictionary *cloudOptions = [NSDictionary dictionaryWithObjectsAndKeys:
        (id)kCFBooleanTrue, NSMigratePersistentStoresAutomaticallyOption, 
        (id)kCFBooleanTrue, NSInferMappingModelAutomaticallyOption, 
        MCCloudMainStoreFileName, NSPersistentStoreUbiquitousContentNameKey,
        self.cloudStoreURL, NSPersistentStoreUbiquitousContentURLKey, 
        nil];

An NSPersistentStoreCoordinator is created, and the non-ubiquitious store added.

    NSPersistentStoreCoordinator *coordinator = [[NSPersistentStoreCoordinator alloc] initWithManagedObjectModel:self.managedObjectModel];
    id oldStore = [coordinator addPersistentStoreWithType:NSSQLiteStoreType configuration:nil URL:oldStoreURL options:localOnlyOptions error:&error];
    if ( !oldStore ) {
        // Move back the old store and present the error
        [fileManager moveItemAtURL:oldStoreURL toURL:storeURL error:NULL];
        [[NSApplication sharedApplication] presentError:error];
        return;
    }

The migratePersistentStore:toURL:options:withType:error: method is used to carry out the migration to the ubiquitous store, which is located at the original store URL.

    // Migrate existing (old) store to new store
    if ( ![coordinator migratePersistentStore:oldStore toURL:storeURL options:cloudOptions withType:NSSQLiteStoreType error:&error] ) {
        [fileManager removeItemAtURL:storeURL error:NULL];
        [fileManager moveItemAtURL:oldStoreURL toURL:storeURL error:NULL];
        [[NSApplication sharedApplication] presentError:error];
        return;
    }

If all goes well, the old non-ubiquitious store is deleted.

    else {
        [[NSFileManager defaultManager] removeItemAtURL:oldStoreURL error:NULL];
    }
}

Care should be taken to ensure no data is lost in this process. It might even be better not to delete the old store outright, but to simply leave it in place or move it to the trash, so that if a problem arises, the user has some recourse to retrieve the old data.

What’s Next…

iCloudCoreDataTester is already useful to test various syncing scenarios. You should install it on two Macs, or virtual machines, and try to get it syncing.

At this point, it is not very robust. In particular, it has no facilities for handling stoppages, so if you remove the iCloud container on one machine, you will probably find the app crashes on the other machine after iCloud has synced the change.

In the next post, I’ll introduce a so-called sentinel file, which will be used to notify the app of unexpected changes to the iCloud container, and allow it to respond graciously, rather than just crashing.

Under the Sheets with iCloud and Core Data: Seeding iCloud

Sat, 26 May 2012 05:38:00 -0400

Before you think about the fine details of syncing your Core Data via iCloud, you need to answer some big questions: How are you going to enable and disable iCloud in your app, and how are you going to seed iCloud with data when syncing gets enabled?

In an ideal world, it would work like this

You bring a new app to market
An end-user installs the app on one device with iCloud activated
The app creates a new store with ubiquity options enabled
Core Data writes all changes to the iCloud container
The end-user installs the app on a second device with iCloud active
The app imports transaction logs from the first device
The user never turns off iCloud on any device
The user never removes the iCloud Data for your app
The user never switches iCloud accounts

Unfortunately, this utopia is far from the reality that most developers will have to face. There are lots of corner cases, and a reasonable chance that you could end up being the dunce. For example,

What if the user disables iCloud for a while?
What if the user wipes iCloud data for the app?
What if they switch iCloud accounts for a bit, perhaps to allow someone else to lend their device?
What if your app is not new, but has existed for some time, and the user could have existing data, perhaps across multiple devices?
What if your app is not new, and you have a pre-existing method of sync (eg, Wi-Fi, Dropbox, MobileMe), such that — unbeknownst to iCloud — there is logically-equivalent data on multiple devices.

When dealing with user data, you can’t be blaise. As an argument, “it seems unlikely, so let’s just ignore these issues” is not going to cut it.

iCloud Does Not Repeat Itself

When making a decision about how to handle your iCloud setup, it is important to grasp a fundamental implementation detail of iCloud: it will only move a piece of data in or out at most once on any device — it never repeats.

For example, if someone is entering or making changes on a device where iCloud is turned off, any changes they make will not generate change logs, and those changes will be lost forever to iCloud. They will not automatically appear when iCloud is switched back on.

Also, once Core Data has imported some iCloud data on a device, that data will not be imported again, even if a new local store is created. You might think that if you delete the local store, then bring up the Core Data stack, it would see that the store is empty and reimport all changes from iCloud. That does not happen. As far as Core Data is concerned, the store already has the changes, and your store will remain empty.

Stoppages

I’ve already hinted that changes made after iCloud is disabled, either via a switch in your app, or via the System Preferences, are permanently lost as far as syncing is concerned. This has some important consequences for your app.

For a start, any objects inserted while iCloud is disabled, and other changes made to the data, will not appear on other devices, even after iCloud is re-engaged. That means the user will see two distinct sets of data. They will wonder why the objects they see on one device are not appearing on the other. But that will likely be the least of the problems…

With inconsistent data sets on different devices, transaction logs may no longer import properly. Imagine that an object was created on Device A while iCloud was disabled, and saved. The user then enables iCloud again, and makes a change to the object. Device B will now receive a transaction log for an update to an object which — as far as it is concerned — does not exist. Import of the log will fail, including any other changes that happen to be committed during that save transaction.

The more entangled the ‘missing’ objects are with the rest of your object graph, the more likely they will be involved in an update, and cause transaction log imports to fail. To the user, it just looks like iCloud is not syncing properly.

Stoppages are bad news, even if you happen to have an app that otherwise fits perfectly with the utopian workflow described earlier. The only way to get the user’s data consistent again is to destroy iCloud’s data container, allow one device to refill it with data, remove local stores on other devices, and allow them to re-import the new data from the iCloud container.

iCloud Seeding

If the picture painted above doesn’t seem very rosy, what can you do about it? The first decision you will need to make is how you will handle data inconsistencies, and what your policy will be for seeding iCloud with data when they arise. You need a way to coordinate devices such that eventually they are all working on the same logical data set.

To try to make the decision easier, I’m going to layout the options as explicitly as possible in the coming sections.

Vanilla Seeding

The first option is to follow Apple’s basic prescription entirely. You simply setup all your stores with the ubiquitous options enabled. If your app is new, users don’t have pre-existing data, and they never log out of iCloud, this should work quite well.

But if a stoppage occurs, and syncing becomes unreliable, the fix requires considerable manual user intervention, and is not very palatable. To end up with consistent data across all devices, the user will have to

Quit/Kill the app on all devices
Delete the app’s iCloud container via System Preferences on all devices
Remove the app’s local data store on all but one device
Launch the app on the device with the intact local store, allowing it to seed the iCloud container.

When the app is next launched on the other devices, it should import the newly seeded data, and all devices should have a consistent set. But this is far from user friendly.

Sophie’s Choice Seeding

If you don’t want your users to have to go through this, or you have a more complex scenario where there may be existing data on multiple devices, you will probably need a more sophisticated approach. In the Sophie’s Choice approach, whenever inconsistencies arise, the user will get to choose whether they want to keep the data from the cloud, or the data from the local store. This should always result in a consistent data set across devices.

It is not quite as straightforward as just giving the user a free choice of data set though. If the device has not previously been syncing, and thus has no transaction logs in the iCloud container, they can indeed choose to keep either the local or the cloud data. But if the device has been syncing, and has logs in iCloud, the only option is to replace the iCloud data with the local data set. This is because of the constraints discussed earlier, namely that Core Data will not reimport data it has already imported from iCloud. Deleting the local store and re-enabling ubiquitous options will not lead to all of the iCloud data being imported.

The Sophie’s Choice variant of seeding requires that your app knows if the device has previously been synced, and if the container has been reset, in order to offer the user the appropriate options when inconsistencies arise. At the time of writing, Core Data gives you no notification of these state changes. In a future post, I will introduce so-called sentinel files, which will reside in the iCloud container, and through continuous monitoring can notify of changes to the set of syncing devices.

Every-Drop-is-Sacred Seeding

If the idea of requiring the user to choose between data sets is not attractive, the last option you have is to attempt to merge data sets whenever a new device is added to iCloud. Unfortunately, if a stoppage occurs, the user will still be forced to fully replace their cloud data in order to guarantee a consistent set of data, but at least it will be possible to merge existing data when devices first start to sync.

One problem with this approach is that you may end up with duplicate data. To fix this, you will want to have a globally unique identifier for your main entities. You can then weed out and delete the duplicates. You will need to do a sweep for duplicates whenever iCloud data is merged.

Migrating Data into iCloud

Getting data into iCloud is generally not as straightforward as just enabling the ubiquitous store options. You have to be very aware of what data is already in iCloud, and which devices have contributed to that data. Once you know that, you may have to explicitly migrate data from a non-ubiquitous store to a ubiquitous store, to allow Core Data to generate transaction logs.

If the iCloud container is empty, such that the current device will be the first to sync, you can simply engage the ubiquity options, and the data will be added to iCloud, no migration needed. Core Data generates a baseline in iCloud, which is effectively a copy of your local store. There has been some discussion as to whether the baseline is actually used, or is just in place for future functionality, but my tests seem to show the baseline is in use. (Update: Although the baseline is used — the initial data is transferred to other devices — future changes to that data do not seem to propagate properly. At this point, I advise against relying on the baseline for seeding initial data.)

The baseline is only generated once, for the first iCloud-enabled device. If you have existing data on other devices, it will have to be migrated in, or it will not be synced between devices. Because of this limitation, you may be better off migrating data in on all devices, even the first to sync.

So how do you actually perform the migration? We will consider this in more detail in the next post, but there are basically two approaches:

Use the persistent store coordinator’s migratePersistentStore:toURL:options:withType:error: method, together with the appropriate ubiquity options, to perform a wholesale migration of the entire non-ubiquitous local store to a ubiquitous one.
Add the ubiquitous and non-ubiquitious stores to the persistent coordinator, and copy objects from one to the other in code.

The later gives you more control over what is imported, but obviously requires more effort.

To Switch or Not to Switch

One question that I have not addressed in this discussion is whether your app should have a switch that allows iCloud syncing to be disabled independent of the global iCloud settings.

If you have very little data to sync, I would suggest not bothering the user with a switch. Just assume that if the user is logged into iCloud, they want the app to sync. On the other hand, if your app needs to sync a lot of data, eg. media files, it is probably best to give the option to disable syncing.

There is a third way: if your app does generate a lot of data, but most of it does not need to be synced, you could include the entities that do need to be synced in one ubiquitous store, and the rest in a non-ubiquitous store. Cross store relationships have to be weak — you have to fetch objects using some identifying property such as a URL or UUID — but this is a relatively small inconvenience.

Next Time…

This has been a very high level discussion of some of the issues you encounter when seeding iCloud with data. Next time, I will introduce a test app, and dig down into the code that you will need. If you want to get an early look, the test app is already on GitHub.

Under the Sheets with iCloud and Core Data: How it Works

Thu, 17 May 2012 11:32:19 -0400

Apple’s documentation for Core Data syncing via iCloud offers very little insight into how it actually works. This is probably quite deliberate: Apple sees this as an implementation detail that developers should not concern themselves with. Unfortunately, if you have no concept of what is happening behind the scenes, at least in vague terms, it is difficult to grasp what you should expect during the syncing process.

To give a concrete example, I was initially under the impression that importing of transaction logs took place at a low level, independent of my application code. This is not actually the case. The import uses the same model classes that the rest of the app uses, and calls the same custom accessor methods. If those accessors cause side effects, such as the creation of new objects, the import fails. Not understanding this ‘implementation detail’ ended up costing me a couple of weeks of head scratching.

We will return to side effects in a future post, but today we will focus on how iCloud syncing works behind the scenes.

Pushing Bits

At its heart, iCloud is not much more than a folder — the iCloud container — that syncs its contents between devices, just like Dropbox. The folder is not readily visible to an end-user — it is hidden away at ~/Library/Mobile Documents — but it is there, and if you drop a file in, it will magically appear on your other iCloud devices.

So how does Core Data fit in? At first you may be tempted to think Core Data is talking to an intelligent server in the cloud, one that understands your entity relationships. This is how Core Data syncing worked under MobileMe. But iCloud syncing is an altogether simpler affair on the server side: the server has no understanding of iCloud at all, it just pushes bits from one device to another.

The intelligence lies entirely in the Core Data framework on each client device. The framework registers changes when your app saves, and writes them away as transaction log files in the iCloud container. iCloud copies the files to the cloud, and then to your other devices. The Core Data framework on these devices notices the new log files, and imports them.

In other words, where with MobileMe there was a single truth database in the cloud, there is now just a collection of transaction logs from different devices, and a number of clients doing their best to independently reconcile those logs. As long as the updates are reasonably distinct, it works well, but when changes overlap considerably, you can get quite tricky situations. A lot of the discussion in this series of posts revolves around working around the conflicts that can arise.

Transaction Logs

When you hear the term transaction log, you may be led to think that it is some low level log file generated by the underlying SQLite database, but the logs are actually just property lists. If you dig into the iCloud container of your app, you will find a number of Core Data Transaction files with the extension .cdt. Each one is a zipped plist file; if you unzip one of them, you should find it contains a file called contents, and this can be opened as a property list in a text editor.

The iCloud container is a subfolder of ~/Library/Mobile Documents; it has the same reverse-DNS name used in the application code, but with all periods replaced by tildes (e.g. ~com~mentalfaculty~mentalcase). Inside the container, you should find one subfolder for each user-device combination. If you dig even deeper into the folder of one of the devices, you will see a folder for each Core Data store that is being synced, and — a few levels deeper — the CDT files themselves.

It is not really necessary to understand the contents of a CDT file, because it is an implementation detail that can change, but in my view it does help to break down some of the mystery of how iCloud—Core Data syncing does its thing, so I want to go through it in some detail.

The structure of the CDT properly list is reasonably transparent. It begins with an array mapping each object in the log file to an entity, and a primary key in the local database.

 {  compressedGlobalIDs = ( "0:0:0", "1:1:0", "2:2:0", "3:3:0", "4:4:0", "5:0:0", "4:0:0", "1:5:0", "0:6:0",
        "0:7:0", "5:8:0", "0:8:0", "5:4:0", "5:9:0", "1:4:0", "2:10:0", "3:11:0", "4:8:0",
        "1:12:0", "0:13:0", "4:14:0", "1:15:0", ...

The first number in each tuple is the index of an entity in the entityNames array.

entityNames = ( "MCNoteFacet", "MCNote", "MCNoteFacetRole", "MCScalableText", "MCFacetPermutation",
    "MCMediaFile", "MCNoteOccurrence", "MCManualCase", "MCCollectionCase", "MCNoteTemplate",
    "MCSlideshow", "MCScheduleManager"
);

The second number in each tuple is the primary key in that entity’s SQLite table. (I have not been able to determine what exactly the third number is for.)

primaryKeys = ( "p1", "p8", "p4", "p32", "p27", "p12", "p48", "p18", "p2", "p46", "p10", "p52", "p9",
    "p15", "p3", "p26", "p45", "p58", "p6", "p5", "p24", "p71", "p11", "p29", "p41", ...

Core Data keeps track of objects across different stores by assigning unique identifiers to inserted objects.

externalDataReferencesInfo = { inserted = ( "1EB4B045-FD76-4791-937B-96BEF29462AD", "BEDDFF4B-6C77-43D3-977E-D1B6734CCFAE", 
"29E8179F-A896-4775-9782-597C2ED92B82", "EF4733C4-C317-41B8-91D8-A4E64BC5D067", ...

Dictionaries are included for all deletions, insertions, and updates. The keys for the dictionaries are the indexes of the corresponding objects in the compressedGlobalIDs array. These indexes are also used to refer to objects that are the target of relationships.

For example, in the insertion dictionary that follows the key is 0, which means the inserted object corresponds to the first item in the compressedGlobalIDs array above. That object has tuple “0:0:0”. The first 0 refers to the index of the entity in the entityNames array, which is MCNoteFacet. The second 0 is the primary key in the MCNoteFacet SQLite table, which is “p1”.

inserted = {
    0 = {
        appearsInSlideshows = 1;
        backgroundColor = <312E3030 30303030 20312E30 30303030 3020312E 30303030 30302031 2E303030 303030>;
        canBeCombinedOnSlide = 0;
        canBePrompt = 0;
        isFactual = 1;
        note = "1";
        noteFacetRole = "2";
        orderIndex = 1;
        promptFacetPermutations = ( );
        responseFacetPermutations = ( "4" );
        scalableText = "3";
        slideFacetOccurrences = ( );
        uniqueId = "2BAC1A44-77AA-466C-A64D-4C0426CCCA07-44703-000132BEDC7A394E";
    };

You can see that the value corresponding to the key is a dictionary of properties, with relationships given as indexes. For example, the property note is a to-one relationship, and has the value “1”. The related object is thus the second object in the compressedGlobalIDs array. To-many relationships are given as arrays of indexes.

(A big thanks to Christian Beer for helping me understand the property list structure.)

Importing Transactions

When import logs are transferred to a device, Core Data attempts to import them into the local store. The exact details of this are very vague. Basically, the process is private, and Apple do not provide any hooks into the import procedure.

It seems that Core Data sets up a private NSPersistentStoreCoordinator, and applies the changes in an associated private NSManagedObjectContext. The managed object context saves, pushing the changes into the store, before posting a notification to alert other contexts to refresh the updated objects. (Thanks to Marcus Zarra for clarifying aspects of this procedure for me.)

You would think, with the import process being a private affair, that your own code could play no role, but it is important to realize that custom accessors and validation methods from your custom NSManagedObject subclasses are used in the background context, and thus do have a very important influence on the success or failure of transaction imports. We will discuss the practicalities of this in a future post.

Under the Sheets with iCloud and Core Data: The Basics

Wed, 16 May 2012 12:51:00 -0400

There can be no doubt that iCloud is an important part of Apple’s future. It’s now less than a year old, but it has been a mixed start. The service itself seems up to the enormous data traffic it has to push, but some aspects of client-side integration are still very rough around the edges.

You don’t have to look far in Apple’s developer forums to know that Core Data syncing is one of those rough spots. An appeal on Twitter, and extensive Googling, led me to just one shipping app that currently offers Core Data syncing over iCloud: Time Butler. I know of no other shipping app that is using it, and certainly no apps from well-known, established developers.

(All eyes are currently on Bare Bones, who have vowed to offer iCloud syncing in their shoebox app Yojimbo before MobileMe syncing is turned off in a few months. Time will tell if they achieve that goal.)

Mental Case

I’ve spent the last 3-4 months integrating Core Data syncing over iCloud into Mental Case for Mac. It is not in the wild yet, but we have started beta testing with a limited audience.

To say it has been a challenge would be an understatement — it has probably been one of the hardest tasks I have ever undertaken for a Mac or iOS app. And so, in an effort to move the state of Core Data/iCloud syncing forward, I have decided to pen a number of posts, in the hope that others will be able to cross the same hurdles faster.

Apple has very little in the way of documentation on Core Data syncing. They do explain the basics of setting up your app, loading your Core Data store, and handling merge notifications, but they don’t go into any detail about how syncing works, and certain scenarios that you will need to deal with in a shipping app. That’s what I plan to cover in these posts.

(Before getting started, it is worth pointing out that Daniel Pasco of Black Pixel delivered an excellent presentation at NSConference 2012, covering many of the issues that I’ll be discussing. You can purchase a video of his talk from the NSConference site.)

Some Restrictions

Let’s start with a few gotchas, some of which wasted days of my time, and are completely undocumented.

Core Data syncing via iCloud does not work in apps using Garbage Collection. This is not documented, it simply does not work. If you have a GC app, time to move to ARC.
iCloud syncing only works for apps sold through the Mac App Store. If you plan to sell your app outside the Mac App Store, you will have to come up with creative ways to enable iCloud syncing. (PDFpen managed to get iCloud syncing enabled for off-store copies by selling a small cloud sync enabler app via the Mac App Store.)
You can only use automatic migration for your iCloud data. If you need to do more extreme migrations, you will need to setup some sort of store versioning, and copy your data to a whole new store.
You cannot use ordered relationships.

If you can cope with these restrictions, you are ready to face iCloud.

Setting Up

In the next few sections, I’ll quickly skim through what Apple covers in its own documentation. This is the bare minimum that you need to get iCloud Core Data syncing working. Unfortunately, it is far from enough to get your app to the production stage.

The first step is to login at the Mac Dev Center and click the link to go to the Developer Certificate Utility. (I will discuss iCloud from the perspective of Mac app development, but much of the information will apply equally to an iOS app.) Select the App IDs, and configure your app to use iCloud. Having done that, you will have to regenerate any provisioning profiles for your app, and install them in Xcode.

Now in Xcode, you should select your app’s target, and in the Summary tab, enable entitlements. You can add one or more iCloud containers here; these correspond to folders in iCloud. The first one should have the same name as the bundle identifier of your app (e.g. com.mentalfaculty.mentalcase.mac). Additional containers can be added with other names. You should use the reverse-DNS naming convention for these containers. Extra containers could be used, for example, to share data between multiple apps, such as when you want to support cross-platform syncing of Macs and iOS devices.

iCloud Containers

In your code, you can request a file URL for your iCloud container using the new NSFileManager method URLForUbiquityContainerIdentifier:. If you only have one container, you can pass in nil. If you have more than one, you need to pass in an identifier.

The identifier is comprised of your Team Identifier Prefix, followed by the container id you entered in the entitlements section (e.g. “<Team ID>.com.mentalfaculty.mentalcase”). So where do you find the Team Identifier Prefix? If you go to the Member Center of the Apple Developer site, and you click the organization link at the top, you should see an entry called ‘Company/Organization ID’. That is the Team Identifier Prefix you need to use.

You should call URLForUbiquityContainerIdentifier: early on in an app’s launch. If it returns nil, it means iCloud is not enabled on the current device. If it returns a file URL, iCloud is active, and the URL gives you a path to the iCloud container, which is a folder in ~/Library/Mobile Documents. Anything you put in there will get synced to other devices using the same container.

Setting Up the Core Data Stack

Having tested whether iCloud is turned on, you are ready to setup your Core Data stack. This is almost the same as when you aren’t using iCloud, with some small differences. If iCloud is on, i.e. if URLForUbiquityContainerIdentifier: returned a URL, you should add extra options for your persistent store.

    NSMutableDictionary *options = [NSMutableDictionary dictionaryWithObjectsAndKeys:
        [NSNumber numberWithBool:YES], NSMigratePersistentStoresAutomaticallyOption,
        [NSNumber numberWithBool:YES], NSInferMappingModelAutomaticallyOption, 
        nil];

    // Add cloud options    
    if ( storeURL ) {
        [options addEntriesFromDictionary:
            [NSDictionary dictionaryWithObjectsAndKeys:
                @"com.mentalfaculty.mentalcase.mainstore.1", NSPersistentStoreUbiquitousContentNameKey,
                storeURL, NSPersistentStoreUbiquitousContentURLKey, 
                nil]];

The value for NSPersistentStoreUbiquitousContentNameKey should be a unique name for the store. It gets mapped to a subfolder in iCloud. The container URL you retrieved above should be passed in with the key NSPersistentStoreUbiquitousContentURLKey.

With the options all ready, you can add your persistent store to the persistent store coordinator.

    dispatch_queue_t backgroundQueue = dispatch_get_global_queue(0,0);
    dispatch_async(backgroundQueue, ^{
        NSError *error;
        [persistentStoreCoordinator lock];
        diskStore = [persistentStoreCoordinator addPersistentStoreWithType:NSSQLiteStoreType configuration:nil URL:localURL options:options error:&error];
        [persistentStoreCoordinator unlock];

The localURL is simply the file URL that you are using for the store. This will usually be in the user’s Library folder somewhere (e.g. Application Support folder).

When you invoke the addPersistentStore… method, it is important to do it on a background thread. With iCloud enabled, this method can block execution for some time, while iCloud retrieves any files it might need from the servers.

With a persistent store coordinator setup, you can now create an NSManagedObjectContext. You probably don’t usually pay much attention to the merge policy it is using, but when integrating with iCloud, you should change the mergePolicy property to something other than the default. A reasonable choice is NSMergeByPropertyObjectTrumpMergePolicy.

Observing Merge Notifications

The last part of a basic iCloud/Core Data setup is to listen for notifications that iCloud has merged changes from another device. Each merge corresponds to a save performed on another device, so it is possible that several notifications fire off in short succession, as iCloud imports the changes one save at a time.

To be notified of merges, simply observe the NSPersistentStoreDidImportUbiquitousContentChangesNotification notification (Is that the longest name in Cocoa?), which is sent by your persistent store coordinator. When you receive the notification, you need to merge the changes into your managed object context in much the same way that you would merge save changes when using multiple contexts.

-(void)persistentStoreCoordinatorDidMergeCloudChanges:(NSNotification *)notification
{
    [managedObjectContext performBlock:^{        
        [managedObjectContext mergeChangesFromContextDidSaveNotification:notification];

You will also need to consider whether managed objects that are retained in your app could be invalidated by a merge, and perhaps need to be re-fetched when the notification fires.

Devil in the Details

That’s basically all that Apple will tell you about Core Data syncing with iCloud, and also all you will tend to find in tutorials on the subject. You could mistakenly be led to believe that integrating iCloud in your Core Data app will be a walk in the park. Unfortunately, at this juncture, nothing could be farther from the truth.

In the coming posts, we will consider the devil in the details, including…

How iCloud Core Data syncing works under the hood.
How you get existing data into iCloud from multiple devices.
Migrating your app from an existing, legacy sync scheme.
What happens when a device has iCloud temporarily disabled.
How you can handle conflicts and validation failures.

Stay tuned.

Fallacies of the Indie Software Business

Mon, 16 Apr 2012 20:12:00 -0400

There’s a lot of talk about upgrade pricing on the Mac App Store. Unfortunately, a lot of it is based on guess-work about how the software market works, by people who don’t actually sell software themselves. Here are some of the bigger fallacies being propagated.

The market is completely efficient. As soon as an app is released, every user has noticed this, and considered whether to purchase it.
Once someone has considered purchasing a piece of software, and rejected it, they will never consider it again.
Only people new to a platform buy software.
A small indie developer can easily saturate their market.
Mac App Store customers are used to paying for software upgrades.
Software developers can only survive without upgrade pricing if the platform has hockey stick growth.
Software developers rely on upgrade spikes to survive, because — at other times — they have no sales.
Software developers earn most of their income from sales spikes associated with an upgrade.
Writing a new app is just as fast and easy as adding features to an existing app, so there is no motivation to upgrade software without upgrade pricing.
Upgrading an app can only generate income via existing users. There are no other advantages to upgrading an app.
It is more useful for a developer to spend their whole life adding features to one app than to develop multiple apps.

Paid Upgrades and the Mac App Store

Wed, 28 Mar 2012 20:12:00 -0400

If you hang around iOS and Mac developer circles enough, you’re sure to hear some complaints about how Apple runs its App Stores, and one of the most common is that they don’t support paid upgrades. Yesterday, Wil Shipley presented the case for paid upgrades on his blog, with some supporting analysis of Delicious Library 2 sales through the years.

Shipley has had success redirecting Apple in the past, most recently with a campaign to introduce Apple-issued developer certificates, which will make their debut via the Gatekeeper feature in Mountain Lion. But on this one, I think he has it wrong, and I doubt very much that Apple will be shifted. Here’s why.

Apple has spent the last 5 years or so cultivating certain expectations in how people purchase digital goods via their stores. If you buy a song in iTunes, it’s yours — forever. If you buy an app in the iOS App Store, it’s yours — forever. And now, if you buy an app in the Mac App Store, it’s yours — (you guessed it) forever. It’s part of what has made Apple so successful in this area. One click purchase, instant download, yours forever.

What Wil is asking Apple to do is to break this implied contract with its customers. Effectively, he wants customers to pay twice for the same product, and I don’t see Apple going down that route. It could actually be hurtful to developers in some ways. One of the reasons people so readily part with their cash in the App Stores is that they know they get to keep what they buy, and even transfer it to any new devices they acquire. They own their software, they don’t rent it.

Wil’s blog post goes into some reasonably detailed analysis of Delicious Library sales trends, concluding that offering free upgrades would result in his company ‘…losing a quarter of our revenue’. Fortunately, his arguments are oversimplified and based on pre-App Store sales models. Some of the analysis is also a bit disingenuous. For example, he shows that for the 18 months after the release of Delicious Library 2, 24% of sales revenue came from upgrades. A closer look shows that most of that probably resulted from the spike immediately following release, and that in fact, taken over the full four years since the app was released, upgrade sales would be much less than 24%.

But even if upgrade sales did account for a quarter of all revenue, it is not valid to conclude that revenue would suffer an equivalent drop if upgrade pricing was not available. The Mac App Store does not offer this stream of income, but does offer other streams that in my experience more than balance any losses. I already mentioned that customers are encouraged to buy apps by the ease with which they can do it in the App Store, and by the knowledge that whatever they purchase becomes a digital possession. It’s difficult to put a number on how large this effect is, but I am convinced by my own app sales that it is significant.

Giving away major upgrades to existing customers has a second positive effect on sales. Because all of your existing customers get the upgrade free, most will download it, and many will tweet about it, and generally encourage friends to take a look at the app. Word of mouth marketing is an enormous boost to any new release, and the more existing customers actually install an upgrade, the better. This could easily make up for the 24% loss in upgrade sales on its own, especially if you take into account the non-linear sales effects you see as you push up the charts.

Shipley implies throughout his post that making major upgrades free would effectively discourage developers from updating their software, because it would not lead to any extra revenue, in the form of a spike in sales.

But, with free major upgrades, we have this unhappy choice: for the next two years we can work on a brand-new product (call it “Delicious D”) or we can work the next version of “Delicious Library ∞”. If we write “Delicious D” we know you’re going to give us another $40 of your money, because it’ll be so awesome. But if we write “Delicious Library ∞” (and we give away major upgrades) then we’re going to get $0 of your money when we’re done.

This is patently false, and any developer who has ever released a major upgrade in the Mac App Store knows it. There is a very large spike in sales, which comes from the combined effect of increased word of mouth from existing customers, featuring by Apple, and increased visibility when the app begins to climb the charts. When I released Mental Case 2 — exclusively on the Mac App Store — I saw an order of magnitude more sales in the days after the release than previous to that. It is simply not true that free upgrades do not generate any additional income.

Ironically, it was Wil who once explained how the pool of potential customers was virtually infinite for any indie developer, that you will never even come close to saturating your market. This is exactly the reason why the App Store works, and does not need to include upgrade pricing. There are always many more shoppers out there than you currently have customers. Nickel and dime-ing your existing customers with paid upgrades means they will resent you, and you are taking your eyes off the bigger prize, the enormous body of potential new customers which should be your target.

Wil Shipley has been making money in this industry longer than most of us, and frankly, I think he is falling into the same trap that the music industry fell into when Napster came to town. What worked yesterday will not necessarily work tomorrow. The game has changed, and Wil wants to keep doing things the way he always has, when his audience is moving on.

By insisting old sales models get incorporated into the App Store, he is prolonging the day when customers forget there ever was such a thing as a paid upgrade, and that is hurting the transition for all of us. Rather than trying to move Apple, he should be adapting his software to the new way of doing things. No paid upgrades, but plenty of upgrade revenue.