如何使用pandas通过多级嵌套将csv转换为json

Question

I've tried to follow a bunch of answers I've seen on SO, but I'm really stuck here. I'm trying to convert a CSV to JSON.

The JSON schema has multiple levels of nesting and some of the values in the CSV will be shared.

Here's a link to one record in the CSV.

Think of this sample as two different parties attached to one document.

The fields on the document (document_source_id, document_amount, record_date, source_url, document_file_url, document_type__title, apn, situs_county_id, state_code) should not duplicate.

While the fields of each entity are unique.

I've tried to nest these using a complex groupby statement, but am stuck getting the data into my schema.

Here's what I've tried. It doesn't contain all fields because I'm having a difficult time understanding what it all means.

j = (df.groupby(['state_code', 
                 'record_date',
                 'situs_county_id',
                 'document_type__title',
                 'document_file_url',
                 'document_amount',
                 'source_url'], as_index=False)
             .apply(lambda x: x[['source_url']].to_dict('r'))
             .reset_index()
             .rename(columns={0:'metadata', 1:'parcels'})
             .to_json(orient='records'))

Here's how the sample CSV should output

{
   "metadata":{
      "source_url":"https://a836-acris.nyc.gov/DS/DocumentSearch/DocumentDetail?doc_id=2019012901225004",
      "document_file_url":"https://a836-acris.nyc.gov/DS/DocumentSearch/DocumentImageView?doc_id=2019012901225004"
   },
   "state_code":"NY",
   "nested_data":{
      "parcels":[
         {
            "apn":"3972-61",
            "situs_county_id":"36005"
         }
      ],
      "participants":[
         {
            "entity":{
               "name":"5 AIF WILLOW, LLC",
               "situs_street":"19800 MACARTHUR BLVD",
               "situs_city":"IRVINE",
               "situs_unit":"SUITE 1150",
               "state_code":"CA",
               "situs_zip":"92612"
            },
            "participation_type":"Grantee"
         },
         {
            "entity":{
               "name":"5 ARCH INCOME FUND 2, LLC",
               "situs_street":"19800 MACARTHUR BLVD",
               "situs_city":"IRVINE",
               "situs_unit":"SUITE 1150",
               "state_code":"CA",
               "situs_zip":"92612"
            },
            "participation_type":"Grantor"
         }
      ]
   },
   "record_date":"01/31/2019",
   "situs_county_id":"36005",
   "document_source_id":"2019012901225004",
   "document_type__title":"ASSIGNMENT, MORTGAGE"
}

Answer 1

You might need to use the json_normalize function from pandas.io.json

from pandas.io.json import json_normalize
import csv
li = []
with open('filename.csv', 'r') as f:
    reader = csv.DictReader(csvfile)
        for row in reader:
          li.append(row)
df = json_normalize(li)

Here , we are creating a list of dictionaries from the csv file and creating a dataframe from the function json_normalize.

Answer 2

Below is one way to export your data:

# all columns used in groupby()
grouped_cols = ['state_code', 'record_date', 'situs_county_id', 'document_source_id'
    , 'document_type__title', 'source_url', 'document_file_url']

# adjust some column names to map to those in the 'entity' node in the desired JSON
situs_mapping = {
    'street_number_street_name': 'situs_street'
,   'city_name': 'situs_city'
,   'unit': 'situs_unit'
,   'state_code': 'state_code'
,   'zipcode_full': 'situs_zip'
}
# define columns used for 'entity' node. python 2 need to adjust to the syntax
entity_cols = ['name', *situs_mapping.values()]
#below for python 2#
#entity_cols = ['name'] + list(situs_mapping.values())

# specify output fields
output_cols = ['metadata','state_code','nested_data','record_date'
    , 'situs_county_id', 'document_source_id', 'document_type__title']

# define a function to get nested_data
def get_nested_data(d):
    return {
        'parcels': d[['apn', 'situs_county_id']].drop_duplicates().to_dict('r')
    ,   'participants': d[['entity', 'participation_type']].to_dict('r')
    }

j = (df.rename(columns=situs_mapping)
    .assign(entity=lambda x: x[entity_cols].to_dict('r'))
    .groupby(grouped_cols)
    .apply(get_nested_data)
    .reset_index()
    .rename(columns={0:'nested_data'})
    .assign(metadata=lambda x: x[['source_url', 'document_file_url']].to_dict('r'))[output_cols]
    .to_json(orient="records")
)

print(j)

Note: If participants contain duplicates and must run drop_duplicates() as we do on parcels , then assign(entity ) can be moved to defining the participants in the get_nested_data() function:

    ,   'participants': d[['participation_type', *entity_cols]] \
           .drop_duplicates() \
           .assign(entity=lambda x: x[entity_cols].to_dict('r')) \
           .loc[:,['entity', 'participation_type']] \
           .to_dict('r')