ios core data Importing large amounts of data

Question

My app needs to import objects from my web API (usually more then 5000 objects) while still interacting with the database. Currently I have the API set to return 100 objects per API request. The problem is that I have a huge performance hit. It can take > 4 seconds to import all the data. Most of this time is spent when calling the save function. I have successfully pushed the import into the background thread. However when I save the main context on the main thread at the end of each api request it can take more then 1 second to save, which pauses all scrolling, animation, etc. What can I do to put the saving of the main context into the background?

Updated: Here is some example code

-(void) importLoop:(NSManagedObjectContext*)mainContext complete:(dispatch_block_t) complete{
    [self apiRequest:^(NSArray *objects) {
        if (objects.count == 0){ // nothing to load.
            dispatch_async(dispatch_get_main_queue(), complete);
            return;
        }
        dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0ul);
        dispatch_async(queue, ^{
            NSManagedObjectContext* bgContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSPrivateQueueConcurrencyType];
            [bgContext setParentContext:mainContext];

            [bgContext performBlock:^{
                for (id object in objects){
                    // import data. This can take > 4 seconds
                    [bgContext save:nil]; // This can take > 1 second
                    // Note: This save merges the data to the main moc... but does not save the data to disk
                }
                [mainContext performBlock:^{
                    [mainContext save:nil]; // This can take > 1 second on main thread
                    [self importLoop:mainContext complete:complete]; // import next batch
                }];
            }];
        });
    }];
}

Update: According to @Wain answer I need to set up the following

mainMoc -> mainThreadMoc -> bgMoc

Where

• mainMoc has Concurrency Type NSPrivateQueueConcurrencyType (this moc is used for saving to disk)
• mainThreadMoc has Concurrency Type NSMainQueueConcurrencyType and is the child of mainMoc
• bgMoc has Concurrency Type NSPrivateQueueConcurrencyType and is the child of mainThreadMoc

Ok now that I have that setup, I have the following problem... when I make changes to mainThreadMoc how do I let bgMoc know about it. I have the following function that is registered for NSManagedObjectContextDidSaveNotification

- (void)managedObjectContextDidSaveNotification:(NSNotification *)notification
{
    NSManagedObjectContext *savedContext = [notification object];

    // ignore change notifications for the main MOC
    if (self.mainMoc == savedContext)
    {
        return;
    }

    if (self.mainMoc.persistentStoreCoordinator != savedContext.persistentStoreCoordinator)
    {
        // that's another database
        return;
    }
    if (savedContext == self.mainThreadMoc){
        [self.mainMoc performBlock:^{
            [self.mainMoc mergeChangesFromContextDidSaveNotification:notification];
        }];
    }else if (savedContext == self.bgMoc){
        [self.mainThreadMoc performBlock:^{
            [self.mainThreadMoc mergeChangesFromContextDidSaveNotification:notification];
        }];
    }
}

Update: I found that with the contexts chained as I mentioned above and if I re create the background context as I do in my initial example then things seem to be working. There is still a noticeable delay when I save the background thread (I presume this is the result of propagating the changes to the main thread). However this is a great improvement in performance.

Answer 1

I would not have started by introducing the extra private-queue parent MOC.

At best it will slightly reduce the total amount of time spent on the main thread while increasing the total wall-clock time to complete the import.

I would also not recommend a third-party framework on top of Core Data while you are still coming to grips with the Core Data framework itself.

As for your question about processing the NSManagedObjectContextDidSaveNotification , if a new bgContext was created temporarily for each ingestion chunk as you show in your first example, there would be no need to deal with those notifications at all because calling save: in a child context immediately and transparently propagates the changes to the parent.

As for the original question and example, calling save: should not take 1 second for 100 objects.

You should not be considering architectural changes (such as different context arrangements) until you understand why it's taking 1 second to save 100 objects.

Hook your app up to instruments and let it tell you what is taking up all the time. The answer may surprise you.

Answer 2

You need to save to the persistent store, not necessarily from the main context.

You should have a private queue context which is parent to your main context and a background context so you can save without impacting the main thread and then merge the changes down to the main context. Main in this refers to main thread, not main ownership.