Set CountVectorizer result to pandas.DataFrame

Question

I need to set pandas.DataFrame with matrix features produced by CountVectorizer.

count_vect = CountVectorizer()
count_vect.fit(text)

xtrain_count = count_vect.transform(train_x)
SaveTxt = pandas.DataFrame()
SaveTxt['text']=xtrain_count

but in the last line SaveTxt['text']=xtrain_count I got following errors!

 raise ValueError('Cannot set a frame with no defined index '
ValueError: Cannot set a frame with no defined index and a value that cannot be converted to a Series

I was wondering how should I set result matrix of CountVectorizer to dataframe? CountVectorizer result is a csr_matrix with about 20000 rows and 200000 columns and contents are integer (1 to 6)

Answer 1

pd.DataFrame(my_csr_matrix.todense())

Here is a proof of concept:

import random

import lorem
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer

m = 10
random.seed(0)
data = [lorem.paragraph() for _ in range(m)]

cv = CountVectorizer()
cv.fit(data)

df = pd.DataFrame(data=cv.transform(data).todense())

print(df.shape)
print(df.head())

Result:

(10, 27)
   0  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26
0  1  2  2  3  3  0  2  0  3  1   2   2   2   1   1   5   3   2   1   3   1   0   2   2   1   4   4
1  0  0  4  1  0  0  1  3  0  3   2   0   1   0   1   1   1   5   3   2   0   0   1   0   0   3   1
2  0  2  3  1  1  1  2  0  2  0   1   1   1   1   1   3   2   0   1   2   1   4   3   0   1   2   5
3  3  3  4  7  1  2  4  2  2  0   1   2   1   1   0   0   0   2   1   3   2   2   2   2   0   3   4
4  2  3  1  2  3  4  1  1  4  3   2   4   2   2   3   3   2   0   2   3   2   5   4   3   2   1   2