使用Python将JSON数据列表导入Postgresql

Question

I'm trying to import a list of JSON Data into the Postgresql using Python but, I'm getting this error

psycopg2.ProgrammingError: can't adapt type 'dict'

when I put the "custom_fields" field with a JSONB[] data type in my created table in the Postgresql Database. Is there any way to fix my problem? Any help would do. Thank you so much

//sample.json

{
    "url": "https://www.abcd.com/",
    "id": 123456789,
    "external_id": null,
    "via": {
        "channel": "email",
        "id": 4, 
        "source": {
            "from": {
            "address": "abc@abc.com", 
            "name": "abc def"
        }, 
        "rel": null, 
        "to": {
            "address": "def@def.com", 
            "name": "def"
            }
        }
    },
    "custom_fields": [
        {
            "id": 234567891,
            "value": "abc_def_ghi"
        },
        {
            "id": 345678912, 
            "value": null
        },
        {
            "id": 456789123, 
            "value": null
        }
    ]
},

{
    "url": "http://wxyz.com/", 
    "id": 987654321, 
    "external_id": null, 
    "via": {
        "channel": "email",
        "id": 4,
        "source": {
            "from": {
                "address": "xyz@xyz.com", 
                "name": "uvw xyz"
            }, 
            "rel": null, 
            "to": {
                "address": "zxc@zxc.com", 
                "name": "zxc"
            }
        }
    },
    "custom_fields": [
        {
            "id": 876543219, 
            "value": "zxc_vbn_asd"
        }, 
        {
            "id": 765432198, 
            "value": null
        }, 
        {
            "id": 654321987, 
            "value": null
        }
    ]   
}

//my_code.py

import json
import psycopg2

connection = psycopg2.connect("host=localhost dbname=sample user=gerald password=1234")
cursor = connection.cursor()

data = []
with open('sample.json') as f:
    for row in f:
        data.append(json.loads(row))

fields = [
    'url',
    'id',
    'external_id',
    'via',
    'custom_fields'
]

for item in data:
    my_data = [item[field] for field in fields]
    for key, value in enumerate(my_data):
        if isinstance(value, dict):
            my_data[key] = json.dumps(value)
    insert_query = "INSERT INTO crm VALUES (%s, %s, %s, %s, %s) "
    cursor.execute(insert_query, tuple(my_data))

connection.commit()
connection.close()

Answer 1

Frameworks like SQLAlchemy would be my goto thing for operations like this.

JSON payload in above example is nested. You'll have to ensure that respective target DB in your DB reflect the same schema.

An example FYR:

from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy_declarative import  Base, Review

engine = create_engine('sqlite:///sqlalchemy_try.db') #you might want to tweak this dialect to db of you choice. 
Base.metadata.bind = engine
DBSession = sessionmaker(bind=engine)

session = DBSession()
with open('sample.json') as f:
    data=f.read()
    jsondata=json.loads(data)
    r = Review(jsondata['url'], int(jsondata['id']), jsondata['external_id'], jsondata['via']['channel']))
    session.add(r)
session.commit()