分析来自 Pandas Dataframe 的代币数据

Question

I'm a relative python noob and also new to natural language processing (NLP).

I have dataframe containing names and sales. I want to: 1) break out all the tokens and 2) aggregate sales by each token.

Here's an example of the dataframe:

name    sales
Mike Smith  5
Mike Jones  3
Mary Jane   4

Here's the desired output:

token   sales
mike    8
mary    4
Smith   5
Jones   3
Jane    4

Thoughts on what to do? I'm using Python.

Answer 1

Assumption : you have a function tokenize that takes in a string as input and returns a list of tokens

I'll use this function as a tokenizer for now:

def tokenize(word):
    return word.casefold().split()

Solution

df.assign(tokens=df['name'].apply(tokenize)).explode('tokens').groupby('tokens')['sales'].sum().reset_index()

In [45]: df
Out[45]:
             name  sales
0      Mike Smith      5
1      Mike Jones      3
2       Mary Jane      4
3  Mary Anne Jane      1

In [46]: df.assign(tokens=df['name'].apply(tokenize)).explode('tokens').groupby('tokens')['sales'].sum().reset_index()
Out[46]:
  tokens  sales
0   anne      1
1   jane      5
2  jones      3
3   mary      5
4   mike      8
5  smith      5

Explanation

1. The assign step creates a column called tokens that applies the tokenize functio

Note : For this particular tokenize function - you can use df['name'].str.lower().str.split() - however this won't generalize to custom tokenizers hence the .apply(tokenize)

this generates a df that looks like

             name  sales              tokens
0      Mike Smith      5       [mike, smith]
1      Mike Jones      3       [mike, jones]
2       Mary Jane      4        [mary, jane]
3  Mary Anne Jane      1  [mary, anne, jane]

2. use df.explode on this to get

             name  sales tokens
0      Mike Smith      5   mike
0      Mike Smith      5  smith
1      Mike Jones      3   mike
1      Mike Jones      3  jones
2       Mary Jane      4   mary
2       Mary Jane      4   jane
3  Mary Anne Jane      1   mary
3  Mary Anne Jane      1   anne
3  Mary Anne Jane      1   jane

3. last step is just a groupy-agg step.

Answer 2

You can use the str.split() method and keep item 0 for the first name, using that as the groupby key and take the sum, then do the same for item -1 (last name) and concatenate the two.

import pandas as pd
df = pd.DataFrame({'name': {0: 'Mike Smith', 1: 'Mike Jones', 2: 'Mary Jane'},
 'sales': {0: 5, 1: 3, 2: 4}})


df = pd.concat([df.groupby(df.name.str.split().str[0]).sum(),
    df.groupby(df.name.str.split().str[-1]).sum()]).reset_index()

df.rename(columns={'name':'token'}, inplace=True)

Answer 3

df[["fname", "lname"]] = df["name"].str.split(expand=True) # getting tokens,considering separated by space

tokens_df = pd.concat([df[['fname', 'sales']].rename(columns = {'fname': 'tokens'}),
                       df[['lname', 'sales']].rename(columns = {'lname': 'tokens'})])

pd.DataFrame(tokens_df.groupby('tokens')['sales'].apply(sum), columns=['sales'])