iOS FMDB获取约三百万个项目：由于内存问题而终止

Question

Recently I am working on an app about sqlite. I used FMDB (2.6.2) and iOS is 10.2.0 . There are about three million items in an database. And every item has three colomns: id name logo .

When showing all the data on UI, the requiremens of the app is blow:

• You can not use the way of fetching more data (just like use UITableView or UICollectionView scroll to bottom to load more,but you still can use UITableView or UICollectionView )
• UI must be very smooth when you scroll
• Don't cause memory overflow

According to the requirments, I test some conditions.

I can satisfy the first and sceond requirement. But the last one ,I find when you fetch all the three million items from db once or many times, it still can cause memory overflow.

Here is my code to fetch data in background queue：

dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{

    while (_usersArrM.count < _countNum)
    {
        NSUInteger offset = _usersArrM.count + usersFromDBOffsetCount;
        if (offset > _countNum)
        {
            offset = _countNum - _usersArrM.count;
        }
        else
        {
            offset = usersFromDBOffsetCount;
        }

        NSString  *sql = [NSString stringWithFormat:@"select * from user limit %zd, %zd", _usersArrM.count, offset];

        [_queue inDatabase:^(FMDatabase *db) {

            FMResultSet *userModelRS = [db executeQuery:sql];
            while (userModelRS.next)
            {
                AndyUserModel *userModel = [[AndyUserModel alloc] init];

                userModel.Id = [userModelRS intForColumn:@"id"];
                userModel.name = [userModelRS stringForColumn:@"name"];
                userModel.logo = [userModelRS stringForColumn:@"logo"];
                [_usersArrM addObject:userModel];
            }

            [userModelRS close];
        }];

        [_queue close];
    }
});

Tips:

• _usersArrM : a NSMutableAarray the datasource of UITableView
• _countNum : the nums of items in db
• usersFromDBOffsetCount : static NSUInteger const usersFromDBOffsetCount = 2000;
• _queue : _queue = [FMDatabaseQueue databaseQueueWithPath:dbPath];
• db download: http://okufme2fh.bkt.clouddn.com/test.db.zip

---

I guess the issue that causes memory overflow is _queue , even though [_queue close] , some memory about db can`t be released correctly . And the reult is when loop some times, app terminate due to memory issue.

Is there one way to release the memory of db completely when one loop is finished . Or what is the correct way to satisfy these requirements for this app.

Thanks.

Answer 1

At some point, no matter how good your data framework is, sheer memory allocation will crash any hardware not powerful enough to handle it. You could try hollowing out the data you're trying to retrieve (ie

SELECT specific_col_1, specific_col_2, ... FROM table WHERE conditions

where specific_col_i is a low memory consumption value on average.

Another safeguard would be to use OFFSET and LIMIT to limit the amount of data you pull from the database at any given time:

SELECT specific_col_1, specific_col_2, ... FROM table WHERE conditions LIMIT x OFFSET j

where x is a safe number of records to pull and process at one time, and j is the index of the last number of rows retrieved. You may have to "play" with the values of x and j .

You will likely have to use an intelligent looping algorithm to selectively 'bring in and process' all of those records. And, you could store the batches you get from the loops off to the side in another table/database if that is needed (not sure exactly what your data processing needs are as data are being brought in or after data are brought in).

Lastly, you may find LabQLite helpful. This is a library I've developed for handling SQLite in an iOS app. LabQLite Model Generator is a model generator I created that would allow you to treat your tables and views as classes in Objective-C (or Swift if you do interop code)). LabQLite also has plenty of mid- and low-level database controller methods that might help with refining the looping/batching process I mentioned above.