文件清单的tfidf

Question

I have a list of documents(TDT2 corpus) and I want to get a vocabulary from it using tfidf. Using textblob is taking forever and I don't see it producing a vocabulary before 5-6 days given the speed. Is there any other technique to go about this? I came across scikit-learn's tfidf technique but I am afraid it too will take the same amount of time.

    from sklearn.feature_extraction.text import CountVectorizer

    results = []
    with open("/Users/mxyz/Documents/wholedata/X_train.txt") as f:
        for line in f:
            results.append(line.strip().split('\n'))

    blob=[]
    for line in results:
        blob.append(line)


    count_vect= CountVectorizer()


   counts=count_vect.fit_transform(blob)
   print(counts.shape)

This keeps producing an error about not accepting a list and that list does not have lower.

Answer 1

I assume results should just be a list , not a list of list s? If that's the case, change:

results.append(line.strip().split('\n'))

to:

results.extend(line.strip().split('\n'))

append is adding the whole list returned by split as a single element in the results list ; extend is adding the items from that list to results individually.

Side-note: As written

blob=[]
for line in results:
    blob.append(line)

is just doing a shallow copy of results the slow way. You can replace that with either blob = results[:] or blob = list(results) (the latter is slower, but if you didn't know what sort of iterable results was and needed it to be a list and nothing else, that's how you'd do it).