有什么方法可以在包含列表的pandas Dataframe中扩展列，并从列表值本身获取列名称？

Question

I've converted a nested JSON file to a pandas DataFrame. Some of the columns now contain lists. They look like this:

0         [BikeParking: True, BusinessAcceptsBitcoin: Fa...
1         [BusinessAcceptsBitcoin: False, BusinessAccept...
2         [Alcohol: none, Ambience: {'romantic': False, ...
3         [AcceptsInsurance: False, BusinessAcceptsCredi...
4         [BusinessAcceptsCreditCards: True, Restaurants...
5         [BusinessAcceptsCreditCards: True, ByAppointme...
6         [BikeParking: True, BusinessAcceptsCreditCards...
7         [Alcohol: none, Ambience: {'romantic': False, ...
8                        [BusinessAcceptsCreditCards: True]
9         [BikeParking: True, BusinessAcceptsCreditCards...
10                                                     None
.
.
.
144070    [Alcohol: none, Ambience: {'romantic': False, ...
144071    [BikeParking: True, BusinessAcceptsCreditCards...
Name: attributes, dtype: object

and this:

0         [Monday 11:0-21:0, Tuesday 11:0-21:0, Wednesda...
1         [Monday 0:0-0:0, Tuesday 0:0-0:0, Wednesday 0:...
2         [Monday 11:0-2:0, Tuesday 11:0-2:0, Wednesday ...
3         [Tuesday 10:0-21:0, Wednesday 10:0-21:0, Thurs...
4                                                      None

144066                                                 None
144067    [Tuesday 8:0-16:0, Wednesday 8:0-16:0, Thursda...
144068    [Tuesday 10:0-17:30, Wednesday 10:0-17:30, Thu...
144069                                                 None
144070    [Monday 11:0-20:0, Tuesday 11:0-20:0, Wednesda...
144071    [Monday 10:0-21:0, Tuesday 10:0-21:0, Wednesda...
Name: hours, dtype: object

Is there any way for me to automatically extract the tags (BikeParking, AcceptsInsurance etc.) and use them as column names while filling the cells with the true/false values. For the Ambience dict I want to do something like Ambience_romantic and true/false in the cells. Similarly, I want to extract the days of the week as Column names and use the hours to fill the cells.

Or is there a way to flatten the json data before? I have tried passing the json data line by line to json_normalize and creating a dataframe from the output but it produces the same result. Maybe I'm doing something wrong?

Format of Original json (yelp_academic_dataset_business.json):

 {
    "business_id":"encrypted business id",
    "name":"business name",
    "neighborhood":"hood name",
    "address":"full address",
    "city":"city",
    "state":"state -- if applicable --",
    "postal code":"postal code",
    "latitude":latitude,
    "longitude":longitude,
    "stars":star rating, rounded to half-stars,
    "review_count":number of reviews,
    "is_open":0/1 (closed/open),
    "attributes":["an array of strings: each array element is an attribute"],
    "categories":["an array of strings of business categories"],
    "hours":["an array of strings of business hours"],
    "type": "business"
}

My inital attempt with json_normalize:

with open('yelp_academic_dataset_business.json') as f:
        #Normalize the json data to flatten it and store output in a dataframe
        frame= json_normalize([json.loads(line) for line in f])

        #write the dataframe to a csv file
        frame.to_csv('yelp_academic_dataset_business.csv', encoding='utf-8', index=False)

What I'm currently trying:

with open(json_filename) as f:
    data = f.readlines()

    # remove the trailing "\n" from each line
    data = map(lambda x: x.rstrip(), data)

    data_json_str = "[" + ','.join(data) + "]"  

    df = read_json(data_json_str)
    #Now Looking to expand df['attributes'] and others here

And I should also mention my aim is to convert it to .csv to load it into a Database. I don't want lists in my database columns.

You can get the original json data from the Yelp Dataset Challenge site: https://www.yelp.ca/dataset_challenge/dataset

Answer 1

You're trying to convert "documents" (semi-structured data) into a table. This could be problematic if one record contains eg 100 attributes which no other records have--you probably don't want to add 100 columns to a master table and have empty cells for all other records.

But in the end you have explained that you intend to do this:

1. Load JSON.
2. Convert to Pandas.
3. Export CSV.
4. Import into a database.

And I am here to tell you that this is all entirely pointless. Mashing the data through all these intermediate formats will only cause problems.

Instead, let's get back to basics:

1. Load JSON.
2. Write to database.

Now the first step is coming up with a schema. Or, if you're using a NoSQL database, you can directly load the JSON with no other steps required.