如何从JSON列表列表创建padas.DataFrame

Question

I have pandas DataFrame from CSV ( gist with small sample ):

|  title   |                       genres               |
--------------------------------------------------------
| %title1% |[{id: 1, name: '...'}, {id: 2, name: '...'}]|
| %title2% |[{id: 2, name: '...'}, {id: 4, name: '...'}]|
...
| %title9% |[{id: 3, name: '...'}, {id: 9, name: '...'}]|

Each title can be associated with a various count of the genres (more or greater 1).

The task is to convert arrays from genre column into columns and put ones (or True s) for each genre:

|  title   | genre_1 | genre_2 | genre_3 | ... | genre_9 |
---------------------------------------------------------
| %title1% |    1    |    1    |    0    | ... |    0    |
| %title2% |    1    |    0    |    0    | ... |    0    |
...
| %title9% |    0    |    0    |    1    | ... |    1    |

Genres are the constant set (about 20 items in that set).

Naive method is:

1. Create the set of all genres
2. Create columns for each genre filled with 0
3. For each row, in the DataFrame check if some of the genres are in the genres columns and fill the column for that genre with 1.

This approach looks a bit weird.

I think that pandas have a more suitable method for that.

Answer 1

As far as I know, there is no way to perform JSON-deserialization on a Pandas dataframe in a vectorized fashion. One way you ought to be able to do this is with .iterrows() which will let you do this in one loop (albeit slower than most built-in pandas operations).

import json

df = # ... your dataframe

for index, row in df.iterrows():
    # deserialize the JSON string
    json_data = json.loads(row['genres'])

    # add a new column for each of the genres (Pandas is okay with it being sparse)
    for genre in json_data:
        df.loc[index, genre['name']] = 1  # update the row in the df itself

df.drop(['genres'], axis=1, inplace=True)

Note that empty cells with be filled with NaN , not 0 -- you should use .fillna() to change this. A brief example with a vaguely similar dataframe looks like

In [1]: import pandas as pd

In [2]: df = pd.DataFrame([{'title': 'hello', 'json': '{"foo": "bar"}'}, {'title': 'world', 'json': '{"foo": "bar", "ba
   ...: z": "boo"}'}])

In [3]: df.head()
Out[3]:
                           json  title
0                {"foo": "bar"}  hello
1  {"foo": "bar", "baz": "boo"}  world

In [4]: import json
   ...: for index, row in df.iterrows():
   ...:     data = json.loads(row['json'])
   ...:     for k, v in data.items():
   ...:         df.loc[index, k] = v
   ...: df.drop(['json'], axis=1, inplace=True)

In [5]: df.head()
Out[5]:
   title  foo  baz
0  hello  bar  NaN
1  world  bar  boo

Answer 2

If your csv data looks like this.

(i added the quotes to the keys of genres json just to work easily with json package. Since it is not the main problem you can do that as preprocessing)

You will have to iterate through all the rows of input DataFrame .

for index, row in inputDf.iterrows():
    fullDataFrame = pd.concat([fullDataFrame, get_dataframe_for_a_row(row)])

in get_dataframe_for_a_row function:

• prepare a DataFrame with column title and value row['title']
• add columns with names formed by appending id to 'genre_'.
• assign them value of 1

and then build a DataFrame for each row and concat them to a full DataFrame . pd.concat() concatenates the dataframe obtained from each row. will merge the comumns if already exist.

finally, fullDataFrame.fillna(0) to replace NaN with 0

your final DataFrame will look like this.

here is the full code:

import pandas as pd
import json

inputDf = pd.read_csv('title_genre.csv')

def labels_for_genre(a):
    a[0]['id']
    labels = []
    for i in range(0 , len(a)):
        label = 'genre'+'_'+str(a[i]['id'])
        labels.append(label)
    return labels

def get_dataframe_for_a_row(row): 
    labels = labels_for_genre(json.loads(row['genres']))
    tempDf = pd.DataFrame()
    tempDf['title'] = [row['title']]
    for label in labels:
        tempDf[label] = ['1']
    return tempDf

fullDataFrame = pd.DataFrame()
for index, row in inputDf.iterrows():
    fullDataFrame = pd.concat([fullDataFrame, get_dataframe_for_a_row(row)])
fullDataFrame = fullDataFrame.fillna(0)

Answer 3

Full working solution without iterrows :

import pandas as pd
import itertools
import json

# read data
movies_df = pd.read_csv('https://gist.githubusercontent.com/feeeper/9c7b1e8f8a4cc262f17675ef0f6e1124/raw/022c0d45c660970ca55e889cd763ce37a54cc73b/example.csv', converters={ 'genres': json.loads })

# get genres for all items
all_genres_entries = list(itertools.chain.from_iterable(movies_df['genres'].values))

# create the list with unique genres
genres = list({v['id']:v for v in all_genres_entries}.values())

# fill genres columns
for genre in genres:
    movies_df['genre_{}'.format(genre['id'])] = movies_df['genres'].apply(lambda x: 1 if genre in x else 0)