提高从pandas列中提取信息的速度

Question

I have a dataframe with around 200,000 datapoints and a column which looks like this (example for 1 datapoint):

'{"id":342,"name":"Web","slug":"technology/web","position":15,"parent_id":16,"color":6526716,"urls":{"web":{"discover":"http://www.kickstarter.com/discover/categories/technology/web"}}}'

I want to extract information about the name and slug. I did the following:

df["cat"], df["slug"] = np.nan, np.nan

for i in range(0, len(df.category)):
    df["cat"][i] = df.category.iloc[i].split('"name":"')[1].split('"')[0]
    df["slug"][i] = df.category.iloc[i].split('"name":"')[1].split('"')[4]

This works perfectly fine, but it takes around 4 hours. Is there any way to make this faster?

Answer 1

Instead of manipulating a DataFrame directly, try using simple data types and create a dataframe in one go. Another solution other than jezrael's:

import json

cat, slug = [], []

for row in df.category:
    d = json.loads(row)
    cat.append(d['cat'])
    slug.append(d['slug'])

df = pd.DataFrame({'cat': cat, 'slug': slug})

Answer 2

You can do it very efficiently with extract and regular expressions:

df['cat'] = df['category'].str.extract('"name":"([^"]+)"')
df['slug'] = df['category'].str.extract('"slug":"([^"]+)"')

df

The question was about improving speed, so here's the performance comparison (tested on a 100,000 rows sample; see note below):

%%timeit

df['cat'] = df['category'].str.extract('"name":"([^"]+)"')
df['slug'] = df['category'].str.extract('"slug":"([^"]+)"')

309 ms ± 10.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit

cat, slug = [], []
for row in df.category:
    d = json.loads(row)
    cat.append(d['name'])
    slug.append(d['slug'])

df1 = pd.DataFrame({'cat': cat, 'slug': slug})

574 ms ± 6.57 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit

df1 = pd.DataFrame([ast.literal_eval(x) for x in df['category']],
                   index=df.index)[['name','slug']]

5.1 s ± 29 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Note: sample generated with:

x = '{"id":342,"name":"Web","slug":"technology/web","position":15,"parent_id":16,"color":6526716,"urls":{"web":{"discover":"http://www.kickstarter.com/discover/categories/technology/web"}}}'
df = pd.DataFrame({'category': [x]*100000})