Processing JSON in SQL Server — OpenJSON + OpenRowset

Question

Context: I have a file that contains several hundred lines of JSON.The T-SQL below is able to convert a single line of the JSON file into tabular form.

Question: I need some help/pointers on how to process the entire JSON file & feed all the lines into the OPENJSON function. Not too familiar with looping & OPENROWSET in T-SQL

Example of what 1 line of the JSON file looks like:

{"meta":{"total_record_count":4870,"total_pages":98,"current_page":84,"per_page":50},"companies":[{"id":"12205901","url":"https://api.mattermark.com/companies/12205901","company_name":"Robert Frances Group","domain":"rfgonline.com"},{"id":"12206056","url":"https://api.mattermark.com/companies/12206056","company_name":"International Technologists, Inc","domain":"internationaltechnologists.com"},{"id":"12206667","url":"https://api.mattermark.com/companies/12206667","company_name":"NPC CONSULTING","domain":"npcconsult.com"},{"id":"12222689","url":"https://api.mattermark.com/companies/12222689","company_name":"E-MONITORING NETWORKS","domain":"e-monitoring.net"},{"id":"12229305","url":"https://api.mattermark.com/companies/12229305","company_name":"Alphadent Corporation","domain":"alphadent.com"},{"id":"12234070","url":"https://api.mattermark.com/companies/12234070","company_name":"Raja Software Labs","domain":"rajasoftwarelabs.com"},{"id":"12243225","url":"https://api.mattermark.com/companies/12243225","company_name":"GRiDD Technologies","domain":"grdd.net"},{"id":"12243560","url":"https://api.mattermark.com/companies/12243560","company_name":"Lapp Fatch Myers \u0026 Gallagher","domain":"lfmgcpas.com"},{"id":"12243922","url":"https://api.mattermark.com/companies/12243922","company_name":"Bright Hope Designs","domain":"brighthopedesigns.com"},{"id":"12244597","url":"https://api.mattermark.com/companies/12244597","company_name":"Keystone Mining Post","domain":"keystoneminingpost.com"},{"id":"12244702","url":"https://api.mattermark.com/companies/12244702","company_name":"Digiplates Software","domain":"digiplates.com"},{"id":"12244731","url":"https://api.mattermark.com/companies/12244731","company_name":"Talon Computer Associates","domain":"taloncorp.com"},{"id":"12245395","url":"https://api.mattermark.com/companies/12245395","company_name":"M D Technology Svc","domain":"mdtechnologyservices.com"},{"id":"12246115","url":"https://api.mattermark.com/companies/12246115","company_name":"Center Point Assessment Solutions","domain":"centerpointcorp.com"},{"id":"12246313","url":"https://api.mattermark.com/companies/12246313","company_name":"Starwire Corporation","domain":"starwire.com"},{"id":"12246990","url":"https://api.mattermark.com/companies/12246990","company_name":"Bravo IT Consulting","domain":"fabgraphics.com"},{"id":"12247778","url":"https://api.mattermark.com/companies/12247778","company_name":"Mobileer","domain":"mobileer.com"},{"id":"12249050","url":"https://api.mattermark.com/companies/12249050","company_name":"EnCompass Knowledge Systems, Inc.","domain":"encompassknowledge.com"},{"id":"12249682","url":"https://api.mattermark.com/companies/12249682","company_name":"Copley Networks Inc","domain":"copleynetworks.com"},{"id":"12250086","url":"https://api.mattermark.com/companies/12250086","company_name":"Fusion Factor Corporation","domain":"fusionfactor.com"},{"id":"12250109","url":"https://api.mattermark.com/companies/12250109","company_name":"Tal Bahir, Inc.","domain":"pingwin.com"},{"id":"12250548","url":"https://api.mattermark.com/companies/12250548","company_name":"The Satellite Centers","domain":"thesatellitecenters.com"},{"id":"12251703","url":"https://api.mattermark.com/companies/12251703","company_name":"Siris Media, Inc.","domain":"sirismedia.com"},{"id":"12252330","url":"https://api.mattermark.com/companies/12252330","company_name":"Codesly, Inc.","domain":"codesly.com"},{"id":"12252709","url":"https://api.mattermark.com/companies/12252709","company_name":"Decision Sciences Research","domain":"dsra.com"},{"id":"12254047","url":"https://api.mattermark.com/companies/12254047","company_name":"Sdsi Business Systems","domain":"sdsibusiness.com"},{"id":"12256590","url":"https://api.mattermark.com/companies/12256590","company_name":"IMC Computer Consulting","domain":"imcco.net"},{"id":"12258678","url":"https://api.mattermark.com/companies/12258678","company_name":"Advocate Software","domain":"advocatesoftware.com"},{"id":"12258904","url":"https://api.mattermark.com/companies/12258904","company_name":"Developers TCT","domain":"developerstct.com"},{"id":"12261351","url":"https://api.mattermark.com/companies/12261351","company_name":"Branded Entertainment Network (Formerly Plaid Social Labs)","domain":"bengroup.com"},{"id":"12261451","url":"https://api.mattermark.com/companies/12261451","company_name":"Esaag","domain":"esaag.com"},{"id":"12261548","url":"https://api.mattermark.com/companies/12261548","company_name":"Clear Design LLC","domain":"cleardesign.com"},{"id":"12261591","url":"https://api.mattermark.com/companies/12261591","company_name":"Mainbrain Productions","domain":"mainbrain.com"},{"id":"12261702","url":"https://api.mattermark.com/companies/12261702","company_name":"Ideal Computer Svc","domain":"idealcs.com"},{"id":"12263405","url":"https://api.mattermark.com/companies/12263405","company_name":"Wizard Neon","domain":"signwizard.com"},{"id":"12270911","url":"https://api.mattermark.com/companies/12270911","company_name":"Village Kitchen Design","domain":"artefactdesign.com"},{"id":"12270912","url":"https://api.mattermark.com/companies/12270912","company_name":"Datatech IT Solutions","domain":"datatech-it.com"},{"id":"12272698","url":"https://api.mattermark.com/companies/12272698","company_name":"The Integration Works, LLC","domain":"integrationworks.com"},{"id":"12277526","url":"https://api.mattermark.com/companies/12277526","company_name":"AMS Business Consulting Inc.","domain":"amsinfotech.com"},{"id":"12277592","url":"https://api.mattermark.com/companies/12277592","company_name":"Sonic Computer","domain":"soniccomputer.com"},{"id":"12278275","url":"https://api.mattermark.com/companies/12278275","company_name":"Migration Software Systems Ltd.","domain":"migration.com"},{"id":"12278603","url":"https://api.mattermark.com/companies/12278603","company_name":"Accurate Computer Technology","domain":"act-mail.com"},{"id":"12278729","url":"https://api.mattermark.com/companies/12278729","company_name":"Teal Networks, LLC.","domain":"teal.net"},{"id":"12279527","url":"https://api.mattermark.com/companies/12279527","company_name":"DiCarlo Arts Design Studio","domain":"dicarloarts.com"},{"id":"12279766","url":"https://api.mattermark.com/companies/12279766","company_name":"Cory Computer Systems","domain":"corycomputersystems.com"},{"id":"12280538","url":"https://api.mattermark.com/companies/12280538","company_name":"Zyom Inc","domain":"zyom.com"},{"id":"12280665","url":"https://api.mattermark.com/companies/12280665","company_name":"Stonetree Management Company","domain":"stonetreemgt.com"},{"id":"12280714","url":"https://api.mattermark.com/companies/12280714","company_name":"AireContact","domain":"airecontact.com"},{"id":"12281004","url":"https://api.mattermark.com/companies/12281004","company_name":"DSA Technologies, Inc.","domain":"dsatechnologies.com"},{"id":"12281175","url":"https://api.mattermark.com/companies/12281175","company_name":"Conscires Agile Practices","domain":"conscires.com"}],"total_companies":4870,"page":84,"per_page":50}

Block below needs to be modified to open the file, loop over all the lines in the file & feed to OPENJSON function.

IF OBJECT_ID('tempdb..#results') IS NOT NULL DROP TABLE #results

DECLARE @json NVARCHAR(MAX)
--Create a variable that holds 1 line of data from the companies_data.json file
--This T-SQL will be modified to iteratively read the data from the file directly using OPENROWSET
SET @json =  N'{"meta":{"total_record_count":4870,"total_pages":98,"current_page":97,"per_page":50},"companies":[{"id":"13463591","url":"https://api.mattermark.com/companies/13463591","company_name":"Modula4","domain":"modula4.com"},{"id":"13531979","url":"https://api.mattermark.com/companies/13531979","company_name":"Rae Development Corp","domain":"raedevelopment.com"},{"id":"13537660","url":"https://api.mattermark.com/companies/13537660","company_name":"jBASE","domain":"jbase.com"},{"id":"13537769","url":"https://api.mattermark.com/companies/13537769","company_name":"D2Effects LLC","domain":"bitefx.com"},{"id":"13537784","url":"https://api.mattermark.com/companies/13537784","company_name":"eSyncTraining","domain":"esynctraining.com"},{"id":"13541751","url":"https://api.mattermark.com/companies/13541751","company_name":"Business Computing","domain":"bcinc.net"},{"id":"13553270","url":"https://api.mattermark.com/companies/13553270","company_name":"KNIGHTLING INC.","domain":"knightling.com"},{"id":"13560476","url":"https://api.mattermark.com/companies/13560476","company_name":"Netcentric systems","domain":"netcentricsystems.com"},{"id":"13560643","url":"https://api.mattermark.com/companies/13560643","company_name":"Fortium Solutions, LLC","domain":"fortiumsolutions.com"},{"id":"13561893","url":"https://api.mattermark.com/companies/13561893","company_name":"PMAC SERVICES INC","domain":"pmacservices.com"},{"id":"13587256","url":"https://api.mattermark.com/companies/13587256","company_name":"Bayonet Inc.","domain":"mybayonet.com"},{"id":"13590982","url":"https://api.mattermark.com/companies/13590982","company_name":"IQA","domain":"instrumentalqa.com"},{"id":"13593143","url":"https://api.mattermark.com/companies/13593143","company_name":"Atech Consultants","domain":"atechconsultants.com"},{"id":"13601156","url":"https://api.mattermark.com/companies/13601156","company_name":"Csoft Corp","domain":"csoftcorp.net"},{"id":"13602832","url":"https://api.mattermark.com/companies/13602832","company_name":"Gareth, Inc.","domain":"garethinc.com"},{"id":"13604890","url":"https://api.mattermark.com/companies/13604890","company_name":"Xpaseo","domain":"xpaseo.com"},{"id":"13610146","url":"https://api.mattermark.com/companies/13610146","company_name":"Imagine IT","domain":"imagineit.com"},{"id":"13610923","url":"https://api.mattermark.com/companies/13610923","company_name":"HTS Solutions Pvt Ltd","domain":"htssolutions.org"},{"id":"13619836","url":"https://api.mattermark.com/companies/13619836","company_name":"Tgm Software","domain":"tgmsoftware.com"},{"id":"13622956","url":"https://api.mattermark.com/companies/13622956","company_name":"Inaspan LLC","domain":"inaspan.com"},{"id":"13627130","url":"https://api.mattermark.com/companies/13627130","company_name":"Forerunner Systems Inc","domain":"forerunnersystems.com"},{"id":"13628312","url":"https://api.mattermark.com/companies/13628312","company_name":"Advanced Integrated Solutions, Inc.","domain":"aisconsulting.net"},{"id":"13629029","url":"https://api.mattermark.com/companies/13629029","company_name":"Pounce Consulting","domain":"pouncecorp.com"},{"id":"13629537","url":"https://api.mattermark.com/companies/13629537","company_name":"CloudServe Corporation","domain":"cloudservecorp.com"},{"id":"13635102","url":"https://api.mattermark.com/companies/13635102","company_name":"Leivio Technologies","domain":"leivio.com"},{"id":"13636749","url":"https://api.mattermark.com/companies/13636749","company_name":"Alpen Technology Group","domain":"alpentg.com"},{"id":"13636754","url":"https://api.mattermark.com/companies/13636754","company_name":"National Schedule Masters","domain":"tractime.com"},{"id":"13645065","url":"https://api.mattermark.com/companies/13645065","company_name":"RISA Technologies","domain":"risa.com"},{"id":"13646467","url":"https://api.mattermark.com/companies/13646467","company_name":"Enterprise Answers, LLC","domain":"enterpriseanswers.com"},{"id":"13647801","url":"https://api.mattermark.com/companies/13647801","company_name":"Office Automation Centers Inc","domain":"officeautomationcenter.com"},{"id":"13650810","url":"https://api.mattermark.com/companies/13650810","company_name":"Mozgomedia","domain":"mozgomedia.com"},{"id":"13652763","url":"https://api.mattermark.com/companies/13652763","company_name":"2000 Networking","domain":"network2000-hi.com"},{"id":"13654103","url":"https://api.mattermark.com/companies/13654103","company_name":"Vega Imaging","domain":"vega-imaging.com"},{"id":"13654874","url":"https://api.mattermark.com/companies/13654874","company_name":"Astha Inc.","domain":"asthainc.com"},{"id":"13656103","url":"https://api.mattermark.com/companies/13656103","company_name":"Management Applied Programming","domain":"mapinc.com"},{"id":"13657848","url":"https://api.mattermark.com/companies/13657848","company_name":"4Hilton Inc.","domain":"4hilton.com"},{"id":"13658020","url":"https://api.mattermark.com/companies/13658020","company_name":"FCC Group","domain":"fccgroup.com"},{"id":"13658316","url":"https://api.mattermark.com/companies/13658316","company_name":"Wavelength Datacom, Inc.","domain":"wavdata.com"},{"id":"13661074","url":"https://api.mattermark.com/companies/13661074","company_name":"Xintex Corporation","domain":"xintex.com"},{"id":"13663085","url":"https://api.mattermark.com/companies/13663085","company_name":"DCL Media Services","domain":"onlinecopycorp.com"},{"id":"13667881","url":"https://api.mattermark.com/companies/13667881","company_name":"LMSPros","domain":"lmspros.com"},{"id":"13669354","url":"https://api.mattermark.com/companies/13669354","company_name":"Sujansky \u0026 Associates, LLC","domain":"sujansky.com"},{"id":"13669681","url":"https://api.mattermark.com/companies/13669681","company_name":"Felosoft LLC","domain":"felosoft.com"},{"id":"13669759","url":"https://api.mattermark.com/companies/13669759","company_name":"Collaborative Technologies Inc","domain":"collaborationtech.net"},{"id":"13670474","url":"https://api.mattermark.com/companies/13670474","company_name":"Freight Stream Inc","domain":"freightstream.com"},{"id":"13675681","url":"https://api.mattermark.com/companies/13675681","company_name":"eDimension Consulting","domain":"edimensionconsulting.com"},{"id":"13676806","url":"https://api.mattermark.com/companies/13676806","company_name":"LTT Net Solutions","domain":"lttnetsolutions.com"},{"id":"13678223","url":"https://api.mattermark.com/companies/13678223","company_name":"OCRTI Consulting Corp","domain":"ocrti.com"},{"id":"13679946","url":"https://api.mattermark.com/companies/13679946","company_name":"Mac Networks","domain":"macnetworks.com"},{"id":"13681009","url":"https://api.mattermark.com/companies/13681009","company_name":"NetMost Web Technologies","domain":"netmostwebdesign.com"}],"total_companies":4870,"page":97,"per_page":50}';

SELECT
   a.*
INTO #results
FROM OPENJSON(@json)
WITH (  
   total_companies int '$.total_companies',
   page int '$.page',
   per_page int '$.per_page',
   companies nvarchar(max) '$.companies' AS JSON
) j
CROSS APPLY OPENJSON(j.companies)
WITH (
   id nvarchar(10) '$.id',
   url nvarchar(max) '$.url',
   company_name nvarchar(100) '$.company_name',
   domain nvarchar(max) '$.domain'
) a

Example of the intended tabular form that I'm trying to get the JSON file into:

Answer 1

I had to do something similar recently and I've modified that in an effort to help. I created a text file ( big_json.txt ) and copied your provided line in it multiple times for effect.

First thing I did was create two tables. I used generic names, but you get the idea.

-- Create a table to hold the JSON input.
CREATE TABLE [dbo].[JsonDump](
    [my_json] [nvarchar](max) NULL
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
GO

-- Create a table to house the parsed content
CREATE TABLE [dbo].[JsonParsed](
    [id] [int] NULL,
    [url] [varchar](255) NULL,
    [company_name] [varchar](255) NULL,
    [domain] [varchar](255) NULL
) ON [PRIMARY]
GO

Once those have been created, you can begin working with your data.

-- Clear JsonDump
TRUNCATE TABLE dbo.JsonDump;
GO

-- Clear JsonParsed ( only if you don't want to keep what's already there )
TRUNCATE TABLE dbo.JsonParsed;
GO

-- Import ( single column ) JSON
BULK INSERT dbo.JsonDump
FROM 'D:\temp\big_json.txt' -- ( <-- my file, point to your own )
WITH (
    ROWTERMINATOR = '\n'
);
GO

-- Select JSON into dbo.JsonParsed
;WITH json_loop AS (
    SELECT my_json FROM dbo.JsonDump
)
INSERT INTO dbo.JsonParsed (
    id, [url], company_name, domain
)
SELECT
    jsn.id, jsn.[url], jsn.company_name, jsn.domain
FROM json_loop
OUTER APPLY (

    SELECT * FROM OPENJSON( json_loop.my_json, '$.companies' )
    WITH (
        id INT '$.id',
        [url] VARCHAR(255) '$.url',
        company_name VARCHAR(255) '$.company_name',
        domain VARCHAR(255) '$.domain'
    )

) AS jsn

-- Show me the money.
SELECT * FROM dbo.JsonParsed;

Returns ( restricted to TOP 5 )

+----------+-----------------------------------------------+----------------------------------+--------------------------------+
|    id    |                      url                      |           company_name           |             domain             |
+----------+-----------------------------------------------+----------------------------------+--------------------------------+
| 12205901 | https://api.mattermark.com/companies/12205901 | Robert Frances Group             | rfgonline.com                  |
| 12206056 | https://api.mattermark.com/companies/12206056 | International Technologists, Inc | internationaltechnologists.com |
| 12206667 | https://api.mattermark.com/companies/12206667 | NPC CONSULTING                   | npcconsult.com                 |
| 12222689 | https://api.mattermark.com/companies/12222689 | E-MONITORING NETWORKS            | e-monitoring.net               |
| 12229305 | https://api.mattermark.com/companies/12229305 | Alphadent Corporation            | alphadent.com                  |
+----------+-----------------------------------------------+----------------------------------+--------------------------------+

Optionally, if you don't want to store the parsed data in a table, you can simply select it as a resultset vs. doing an INSERT.

;WITH json_loop AS (
    SELECT my_json FROM dbo.JsonDump
)
SELECT
    jsn.id, jsn.[url], jsn.company_name, jsn.domain
FROM json_loop
OUTER APPLY (

    SELECT * FROM OPENJSON( json_loop.my_json, '$.companies' )
    WITH (
        id INT '$.id',
        [url] VARCHAR(255) '$.url',
        company_name VARCHAR(255) '$.company_name',
        domain VARCHAR(255) '$.domain'
    )

) AS jsn

There are other options I'm sure, but this is what worked best for us. Regardless, hopefully this can save you some time.