Pandas 列表中的词频

Question

I have a column of tokenized, lemmatized text in a pandas df. I'm trying to create a matrix of word frequencies so that I can then go on to dimensionality reduction.

I keep running into an error that Python was expecting a string but got a list. TypeError: sequence item 0: expected str instance, list found

I've tried an handful of ways, but run into errors each time. I'm not sure how to account for a list.

Here are a few of the methods I've tried:

Option 1 :

from collections import Counter
df['new_col'] = Counter()
for token in df['col']:
    counts[token.orth_] += 1

This generated ValueError: Length of values does not match length of index

Option 2 :

Counter(' '.join(df['col']).split()).most_common()

Which generated: TypeError: sequence item 0: expected str instance, list found

Option 3 :

pd.Series(values = ','.join([(i) for i in df['col']]).lower().split()).value_counts()[:]

Which generated again: TypeError: sequence item 0: expected str instance, list found

Edit: Sample data:

col
[indicate, after, each, action, step, .]
[during, september, and, october, please, refrain]
[the, work, will, be, ongoing, throughout, the]
[professional, development, session, will, be]

Answer 1

Easy Answer

Given what you've told us, the best solution as 9dogs mentioned is to use scikit-learn's CountVectorizer . I'm making some assumptions here on what format you'd like the data in, but here's what will get you a doc x token dataframe where the values are the counts of the tokens within a document. It assumes that df['col'] is a pandas series where values are lists.

>>> from sklearn.feature_extraction.text import CountVectorizer
>>> cv = CountVectorizer(analyzer=lambda x: x)
>>> counted_values = cv.fit_transform(df['col']).toarray()
>>> df = pd.DataFrame(counted_values, columns=cv.get_feature_names())
>>> df.iloc[0:5, 0:5]
   .  action  after  and  be
0  1       1      1    0   0
1  0       0      0    1   0
2  0       0      0    0   1
3  0       0      0    0   1

CountVectorizer can tokenize for you, and will by default, so we pass an identity lambda function to the analyzer argument to tell it that our documents are pre-tokenized.

Suboptimal Answer

I wouldn't recommend this, but I think it's helpful if you want to understand how Counters work. Since your values are a list, you can use .apply on each row of your series.

>>> counted_values = df['col'].apply(lambda x: Counter(x))
>>> counted_values
0    {'.': 1, 'after': 1, 'indicate': 1, 'action': ...
1    {'during': 1, 'and': 1, 'october': 1, 'please'...
2    {'will': 1, 'ongoing': 1, 'work': 1, 'the': 2,...
3    {'development': 1, 'professional': 1, 'session...
dtype: object

So now you have a series of dicts, which isn't very helpful. You could convert this to a dataframe similar to what we have above with the following:

>>> suboptimal_df = pd.DataFrame(counted_values.tolist())
>>> suboptimal_df.iloc[0:5, 0:5]
     .  action  after  and   be
0  1.0     1.0    1.0  NaN  NaN
1  NaN     NaN    NaN  1.0  NaN
2  NaN     NaN    NaN  NaN  1.0
3  NaN     NaN    NaN  NaN  1.0

I wouldn't recommend this because apply is slow plus it's already a little goofy that we're storing lists as Series values, dicts are equally as goofy. DataFrames do best as structured containers of numeric or string values (think spreadsheets) and not different container types.