改进 EFCore 查询以实现快速操作

Question

I read a sqlite database (The size of the database is about 3 MB, So there is not much information, each table is about 1 or 2 thousand rows) and extract information from it, Then I add this information to a new database. The whole operation takes about 40 seconds.

How can I reduce this time and get the operation done as quickly as possible? (Task, Parallel, async,...)

I am currently using this code:

await Task.Run(async () =>
            {
                var pkgs = new ManifestTable();
                var mydb = new dbContext();
                await mydb.Database.EnsureDeletedAsync();
                await mydb.Database.EnsureCreatedAsync();
                using (var msixDB = new MSIXContext())
                {
                    foreach (var item in await msixDB.IdsMSIXTable.ToListAsync())
                    {
                        var rowId = item.rowid;
                        var manifests = await msixDB.Set<ManifestMSIXTable>().Where((e) => e.id == rowId).ToListAsync();

                        foreach (var manifest in manifests)
                        {
                            pkgs = new ManifestTable();
                            pkgs.PackageId = item.id;


                            var productMap = await msixDB.ProductCodesMapMSIXTable.FirstOrDefaultAsync((e) => e.manifest == manifest.rowid);
                            if (productMap != null)
                            {
                                var prdCode = await msixDB.ProductCodesMSIXTable.FirstOrDefaultAsync((e) => e.rowid == productMap.productcode);
                                if (prdCode != null)
                                {
                                    pkgs.ProductCode = prdCode.productcode;
                                }
                            }
                            var publisherMap = await msixDB.Set<PublishersMapMSIXTable>().FirstOrDefaultAsync((e) => e.manifest == manifest.rowid);

                            if (publisherMap != null)
                            {
                                var publisher = await msixDB.PublishersMSIXTable.FirstOrDefaultAsync((e) => e.rowid == publisherMap.norm_publisher);

                                if (publisher != null)
                                {
                                    pkgs.Publisher = publisher.norm_publisher;
                                }
                            }

                            var pathPart = manifest.pathpart;
                            var yml = await msixDB.PathPartsMSIXTable.FirstOrDefaultAsync((e) => e.rowid == pathPart);
                            if (yml != null)
                            {
                                pkgs.YamlName = yml.pathpart;
                            }

                            var version = await msixDB.VersionsMSIXTable.FirstOrDefaultAsync((e) => e.rowid == manifest.version);
                            if (version != null)
                            {
                                pkgs.Version = version.version;
                            }
                            await mydb.ManifestTable.AddAsync(pkgs);
                        }
                    }
                     await mydb.SaveChangesAsync();
                }

            });

Answer 1

Treating database as object storage is worst idea ever. You have to reduce database roundtrips as it is possible. In your case - by just one request. Also do not play with Task.Run, Parallel, etc. if you do not know which part is slow. In your case - database roundtrips.

var mydb = new dbContext();
await mydb.Database.EnsureDeletedAsync();
await mydb.Database.EnsureCreatedAsync();

using (var msixDB = new MSIXContext())
{
    var query = 
        from item in msixDB.IdsMSIXTable
        from manifest in msixDB.Set<ManifestMSIXTable>().Where(e => e.id == item.rowId)
        from productMap in msixDB.ProductCodesMapMSIXTable.Where(e => e.manifest == manifest.rowid).Take(1).DefaultIfEmpty()
        from prdCode in msixDB.ProductCodesMSIXTable.Where(e => e.rowid == productMap.productcode).Take(1).DefaultIfEmpty();
        from publisherMap in msixDB.Set<PublishersMapMSIXTable>().Where(e => e.manifest == manifest.rowid).Take(1).DefaultIfEmpty()
        from publisher in msixDB.PublishersMSIXTable.Where(e => e.rowid == publisherMap.norm_publisher).Take(1).DefaultIfEmpty()
        from yml in msixDB.PathPartsMSIXTable.Where(e => e.rowid == manifest.pathpart).Take(1).DefaultIfEmpty()
        from version in msixDB.VersionsMSIXTable.Where(e => e.rowid == manifest.version).Take(1).DefaultIfEmpty()
        select new ManifestTable
        {
            PackageId = item.id,
            ProductCode = prdCode.productcode,
            Publisher = publisher.norm_publisher,
            YamlName = yml.pathpart,
            Version = version.version
        };

    mydb.ManifestTable.AddRange(await query.ToListAsync());
    await mydb.SaveChangesAsync();
}

Answer 2

You should start by seeing if there are any algorithmic improvements before trying to do things in parallel etc.

You have two nested loops, so if each table have a few thousands of rows the inner loop body will be running on the magnitude of 10^6, not terrible, but a fair amount.

In the inner loop you are then running a whole bunch of FirstOrDefaultAsync statements. If these are not indexed it will require all rows to be scanned, and this will be slow. So, to start of ensure you have appropriate indices for all the tables. This is done on to ensure that searching for a specific item is in constant time.

You also seem to be doing repeated lookups for PublishersMapMSIXTable with the same parameters. Avoiding unnecessarily repeated operations should be one of the first things to fix, since it is just wasted cycles.

If the whole operation is run on a background thread it is unlikely that all the async calls will help much, it will save a little bit of memory, but cause some bouncing between threads. So if performance if important regular synchronous methods will probably be a little bit faster.

And as always with regards to performance, measure . A good performance profiler should tell you with what most of the time is spent in, and adding some stopwatches is easy if you do not have one. Even very experienced programmers can be completely wrong if they try to guess what the slow parts are.