Function to Parse Arrays in JSON to Insert Into a SQL Table

Question

Trying to take a web API's JSON response and populate a SQL database with the results.

Part of the JSON response has this array:

"MediaLinks": [
            {
                "MediaType": "Datasheets",
                "SmallPhoto": "",
                "Thumbnail": "",
                "Title": "SN54HC374, SN74HC374",
                "Url": "http://www.ti.com/general/docs/suppproductinfo.tsp?distId=10&gotoUrl=http%3A%2F%2Fwww.ti.com%2Flit%2Fgpn%2Fsn74hc374"
            },
            {
                "MediaType": "Product Photos",
                "SmallPhoto": "http://media.digikey.com/photos/Texas%20Instr%20Photos/296-20-DIP_sml.jpg",
                "Thumbnail": "http://media.digikey.com/photos/Texas%20Instr%20Photos/296-20-DIP_tmb.jpg",
                "Title": "20-DIP,R-PDIP-Txx",
                "Url": "http://media.digikey.com/photos/Texas%20Instr%20Photos/296-20-DIP.jpg"
            },
            {
                "MediaType": "Featured Product",
                "SmallPhoto": "",
                "Thumbnail": "",
                "Title": "Logic Solutions",
                "Url": "https://www.digikey.com/en/product-highlight/t/texas-instruments/logic-solutions "
            },
            {
                "MediaType": "Featured Product",
                "SmallPhoto": "",
                "Thumbnail": "",
                "Title": "Analog Solutions",
                "Url": "https://www.digikey.com/en/product-highlight/t/texas-instruments/analog-solutions "
            },
            {
                "MediaType": "PCN Design/Specification",
                "SmallPhoto": "",
                "Thumbnail": "",
                "Title": "Copper Bond Wire Revision A 04/Dec/2013",
                "Url": "http://media.digikey.com/pdf/PCNs/Texas%20Instruments/PCN20120223003A_Copper-wire.pdf"
            },
            {
                "MediaType": "PCN Design/Specification",
                "SmallPhoto": "",
                "Thumbnail": "",
                "Title": "Material Set 30/Mar/2017",
                "Url": "http://media.digikey.com/pdf/PCNs/Texas%20Instruments/PCN20170310000.pdf"
            }
        ],

For testing I've issued the request and then written the response to a file and I'm experimenting with this file to come up with the correct code

conn.request("POST", "/services/partsearch/v2/partdetails", json.dumps(payload), headers)

res = conn.getresponse()
data = res.read()

data_return = json.loads(data)
print(json.dumps(data_return, indent=4))

with open(y["DigiKeyPartNumber"]+".json", "w") as write_file:
    json.dump(data_return, write_file, indent=4, sort_keys=True)
write_file.close()

Then in my test code I've tried this:

import json

with open(r"C:\Users\george\OneDrive\Documents\296-1592-5-ND.json") as json_file:
    data = json.load(json_file)

values = ""
placeholder = '?'
thelist = []
thelist = list(data['PartDetails']['MediaLinks'])
print(type(thelist))
#print(thelist)

placeholders = ', '.join(placeholder for unused in (data['PartDetails']['MediaLinks']))
query = 'INSERT INTO thetable VALUES(%s)' % placeholders
print(query)

But this just produces the following output:

<class 'list'>
INSERT INTO thetable VALUES(?, ?, ?, ?, ?, ?)

For reference this creates what I think will work except for the trailing comma:

if len(data['PartDetails']['MediaLinks']):
    print('The length is: ' + str(len(data['PartDetails']['MediaLinks'])))
    #print(type(data['PartDetails']['MediaLinks']))
    for mediadata in data['PartDetails']['MediaLinks']:
        #print(mediadata)
        for element in mediadata:
            #print(element + ' is "' + mediadata[element] + '"')
            values += '"' + mediadata[element] + '", '
    #print(list(data['PartDetails']['MediaLinks'][1]))
        print(values + "\n")
        values = ""
else:
    print('It is empty')

Which produces this:

The length is: 6
"Datasheets", "", "", "SN54HC374, SN74HC374", "http://www.ti.com/general/docs/suppproductinfo.tsp?distId=10&gotoUrl=http%3A%2F%2Fwww.ti.com%2Flit%2Fgpn%2Fsn74hc374",

"Product Photos", "http://media.digikey.com/photos/Texas%20Instr%20Photos/296-20-DIP_sml.jpg", "http://media.digikey.com/photos/Texas%20Instr%20Photos/296-20-DIP_tmb.jpg", "20-DIP,R-PDIP-Txx", "http://media.digikey.com/photos/Texas%20Instr%20Photos/296-20-DIP.jpg",

"Featured Product", "", "", "Logic Solutions", "https://www.digikey.com/en/product-highlight/t/texas-instruments/logic-solutions ",

"Featured Product", "", "", "Analog Solutions", "https://www.digikey.com/en/product-highlight/t/texas-instruments/analog-solutions ",

"PCN Design/Specification", "", "", "Copper Bond Wire Revision A 04/Dec/2013", "http://media.digikey.com/pdf/PCNs/Texas%20Instruments/PCN20120223003A_Copper-wire.pdf",

"PCN Design/Specification", "", "", "Material Set 30/Mar/2017", "http://media.digikey.com/pdf/PCNs/Texas%20Instruments/PCN20170310000.pdf",

In the table I've created in SQL it uses the same column names as the keys in the JSON array. There are several arrays in the JSON response so I'm hoping to create a generic function that accepts the JSON array and creates the correct SQL INSERT statements to populate the tables with the JSON data. I'm planning on using pyodbc and best case is something that works for both Python 2.7 as well as 3.x

Updated Information:

I found the following code snippet which comes very close:

for thedata in data['PartDetails']['MediaLinks']:
    keys, values = zip(*thedata.items())
    print(values) #This will create the VALUES for the INSERT Statement
print(keys) #This will create the COLUMNS, need to add the PartDetailsId field

I was trying to find a way to get the keys before I ran this for loop because I would have to replace the print statements with the actual SQL INSERT statement.

When I check type(newdata['PartDetails']['MediaLinks']) is returns <class 'list'> in Python 3.7.4 so even though it looks like a dictionary it's treated like a list and .keys() fails to try and grab the keys

Answer 1

Use cursor.executemany() to execute the query on all the rows in the MediaLinks list.

You can't pass the dictionaries directly, though, because iterating over a dictionary returns the keys, not the values. You need to convert this to a list of values, using one of the methods in How to convert list of dictionaries into list of lists

colnames = ", ".join (data['PartDetails']['MediaLinks'][0].keys())
placeholders = ", ".join(["?"] * len(data['PartDetails']['MediaLinks'][0]))
query = "INSERT INTO MediaLInks (" + colnames + ") VALUES (" + placeholders + ")"
cursor.executemany(query, [tuple(d.values()) for d in data['PartDetails']['MediaLinks']])

Answer 2

Just for completeness I want to post a formatted code snippet that is working for me. This would not have been possible without @barmar 's help so thanks again.

The end goal is to convert this into a function so that I can pass in the arrays from a JSON response and have it populate the correct SQL tables with the data. This is close to being complete but not quite there yet.

import pyodbc

conn = pyodbc.connect('Driver={SQL Server};Server=GEORGE-OFFICE3\SQLEXPRESS01;Database=Components;')

cursor = conn.cursor()

with open(r"C:\Users\george\OneDrive\Documents\296-1592-5-ND.json") as json_file:
    data = json.load(json_file)

x = tuple(data['PartDetails']['MediaLinks'][0])
a = str(x).replace("'","").replace("(","")

query = "INSERT INTO MediaLinks (PartDetailsId, " + a + " VALUES(" + str(data['PartDetails']['PartId'])

b = ""
for i in range(len(x)):
    b += ", ?"
b += ")"

query += b

cursor.executemany(query, [tuple(d.values()) for d in data['PartDetails']['MediaLinks']])
cursor.commit()

conn.close()