SQL Server循环openrowset性能

Question

I have the following stored procedure to loop through hundreds of different JSON files that are downloaded to the server every day.

The issue is that the query takes a good 15 minutes to run, I will need to create something similar soon for a larger amount of JSON files, is somebody able to point me in the correct direction in regards to increasing the performance of the query?

DECLARE @json VARCHAR(MAX) = ''
DECLARE @Int INT = 1
DECLARE @Union INT = 0
DECLARE @sql NVARCHAR(max)
DECLARE @PageNo INT = 300

WHILE (@Int < @PageNo)
BEGIN
    SET @sql = (
    'SELECT 
        @cnt = value
    FROM 
        OPENROWSET (BULK ''C:\JSON\tickets' + CONVERT(varchar(10), @Int)  + '.json'', SINGLE_CLOB) as j
        CROSS APPLY OPENJSON(BulkColumn)
    WHERE
        [key] = ''tickets''
    ')
EXECUTE sp_executesql @sql, N'@cnt nvarchar(max) OUTPUT', @cnt=@json OUTPUT

IF NOT EXISTS (SELECT * FROM OPENJSON(@json) WITH ([id] int) j JOIN tickets t on t.id = j.id)
BEGIN
    INSERT INTO
        tickets (id, Field1)
    SELECT
        *
    FROM OPENJSON(@json)
         WITH ([id] int, Field1 int) 
END

END

Answer 1

It seems your BULK INSERT in the loop is the bottleneck. Generally a BULK INSERT is the fastest way to retrieve data. Anyway, here it seems the amount of files is your problem.

To make things faster you would want to read the JSON files in parallel. You could do that by first creating the complete dynamic sql query for all files or maybe for some file groups and simultaneously read.

I would rather advise to use Integration Services with a script component as a source in parallel data flow tasks. First read all files from your destination folder, split them for example in 4 groups, for each group have a loop container that runs in parallel. Depending on your executing machine, you can use as many parallel flows as possible. Allready 2 dataflows should make up for the overhead of integration services.

Another option would be to write a CLR (common language runtime) stored procedure and parallelly deserialize JSON using C#.

It also depends on the machine doing the job. You would want to have enough random-access memory and free cpu power, so it should be considered to do the import while the machine is not busy.

Answer 2

So one method I've had success with when loading data into tables from lots of individual XML files, which you might be able to apply to this problem is by using the FileTable feature of SQL server.

The way it worked was to set up a filetable in the database, then allow access to the FileStream share that was created on the server for the process that was uploading the XML files. XML files were then dropped into the share and were immediately available in the database for querying using xPath.

A process would then run xPath queries would load the required data from the XML into the required tables and keep track of which files had been loaded, then when the next schedule came along, only load data from the newest files.

A scheduled task on the machine would then remove files when they were no longer required.

Have a read up on FileTable here:

FileTables (SQL Server)

It's available in all SQL server editions.