Truncated Core Data NSData objects

Question

I am saving arrays of doubles in an NSData* object that is persisted as a binary property in a Core Data (SQLite) data model. I am doing this to store sampled data for graphing in an iPhone app. Sometimes when there are more than 300 doubles in the binary object not all the doubles are getting saved to disk. When I quit and relaunch my app there may be as few as 25 data points that have persisted or as many as 300.

Using NSSQLitePragmasOption with synchronous = FULL and this may be making a difference. It is hard to tell, as bug is intermittent.

Given the warnings about performance problems as a result of using synchronous = FULL, I am seeking advice and pointers.

Thanks.

[[ Edit: here is code. ]]

The (as yet unrealized) intent of -addToCache: is to add each new datum to the cache but only flush (fault?) Data object periodically.

From Data.m

@dynamic dataSet; // NSData * attribute of Data entity

 - (void) addDatum:(double_t)datum
    {
    DLog(@"-[Data addDatum:%f]", datum);
    [self addToCache:datum];
    }

- (void) addToCache:(double_t)datum
    {
    if (cache == nil)
        {
        cache = [NSMutableData dataWithData:[self dataSet]];
        [cache retain];
        }
    [cache appendBytes:&datum length:sizeof(double_t)];
    DLog(@"-[Data addToCache:%f] ... [cache length] = %d; cache = %p", datum, [cache length], cache);
    [self flushCache];
    }

- (void) wrapup
    {
    DLog(@"-[Data wrapup]");
    [self flushCache];
    [cache release];
    cache = nil;
    DLog(@"[self isFault] = %@", [self isFault] ? @"YES" : @"NO"); // [self isFault] is always NO.
    }

- (void) flushCache
    {
    DLog(@"flushing cache to store");
    [self setDataSet:cache];
    DLog(@"-[Data flushCache:] [[self dataSet] length] = %d", [[self dataSet] length]);
    }

- (double*) bytes
    {
    return (double*)[[self dataSet] bytes];
    }

- (NSInteger) count
    {
    return [[self dataSet] length]/sizeof(double);
    }

- (void) dump
    {
    ALog(@"Dump Data");
    NSInteger numDataPoints = [self count];
    double *data = (double*)[self bytes];
    ALog(@"numDataPoints = %d", numDataPoints);
    for (int i = 0; i

Answer 1

I was trying to get behavior as if my Core Data entity could have an NSMutableData attribute. To do this my NSManagedObject (called Data) had an NSData attribute and an NSMutableData ivar. My app takes sample data from a sensor and appends each data point to the data set - this is why I needed this design.

On each new data point was appended to the NSMutableData and then the NSData attribute was set to the NSMutableData.

I suspect that because the NSData pointer wasn't changing (though its content was), that Core Data did not appreciate the amount of change. Calling -hasChanged on the NSManagedObjectContext showed that there had been changes, and calling -updatedObjects even listed the Data object as having changed. But the actual data that was being written seems to have been truncated (sometimes).

To work around this I changed things slightly. New data points are still appended to NSMutableData but NSData attribute is only set when sampling is completed. This means that there is a chance that a crash might result in truncated data - but for the most part this work around seems to have solved the problem.

Caveat emptor: the bug was always intermittent, so it is possible that is still there - but just harder to manifest.