Python-如何计算不同推文中前100个单词的最高tf-idf值

Question

I have tens of thounds of tweets saved in one .txt file, I want to calculate calculate the highest tf-idf value of the first 100 words in these tweeets, in other words, I want to compare the word's tf-idf value between different tweets, presently,the only thing that I could complete is comparing word's tf-idf value in the same tweets, I cannot find a way to compare word's tf-idf value between different tweets.

Please help me,I have been upset for a long time because of this problem. /(ㄒoㄒ)/~~

Blow is my code:(only able to calculate the term's tfidf value in same tweets)

with open('D:/Data/ows/ows_sample.txt','rb') as f:
    tweet=f.readlines()
lines = csv.reader((line.replace('\x00','') for line in tweet), delimiter=',', quotechar='"')
wordterm=[]
for i in lines:
    i[1]= re.sub(r'http[s]?://(?:[a-z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-f][0-9a-f]))+|(?:@[\w_]+)', "", i[1])
    tweets=re.split(r"\W+",i[1])
    tweets=[w.lower() for w in tweets if w!=""]
    stopwords = open("D:/Data/ows/stopwords.txt", "r").read().split()
    terms = [t for t in tweets if not t in stopwords]
    wordterm.append(terms)

word=[' '.join(t) for t in wordterm]
tfidf_vectorizer = TfidfVectorizer(min_df = 1,use_idf=True)
tfidf_matrix = tfidf_vectorizer.fit_transform(word)
terms_name = tfidf_vectorizer.get_feature_names()
toarry=tfidf_matrix.todense()

#below code will output the tf-idf value of each tweets' terms.
for ii in range(0,len(toarry)):
    print u"第"+ ii +u"个tweets"
    for jj in range(0,len(terms_name)):
        print terms_name[jj],'-',tfidf_matrix[ii,jj]

Answer 1

Now that I understand your question, I will try to answer your question a little better.

To get the top 100 'tf-idf' scores in a way that is comparable across all tweets would either mean that you are letting go of the notion that there are distinct tweets, or you want to be able to compare the same words to each other by tf-idf score.

So for the first scenario, imagine that all your words are in 1 'document'. This would essentially eliminate the 'idf' aspect of tf-idf, and what you'll get is basically a word count vectorizer, which can be compared with one another and you can get the top 100 words this way.

words = ['the cat sat on the mat cat cat']
tfidf_vectorizer = TfidfVectorizer(min_df = 1,use_idf=True)
tfidf_matrix = tfidf_vectorizer.fit_transform(words)
terms_name = tfidf_vectorizer.get_feature_names()
toarry=tfidf_matrix.todense()

toarry:
    matrix([ .75,  0.25,  0.25,  0.25,  0.5])

The other scenario is that you take each tweet separately, and then you compare the scores by their tf-idf scores. This would result in the same words having different scores, because that's what tf-idf does - it calculates the importance of the word in the document relative to the corpus .

words = ['the cat sat on the mat cat', 'the fat rat sat on a mat', 'the bat and a rat sat on a mat']
tfidf_vectorizer = TfidfVectorizer(min_df = 1,use_idf=True)
tfidf_matrix = tfidf_vectorizer.fit_transform(words)
terms_name = tfidf_vectorizer.get_feature_names()
toarry=tfidf_matrix.todense()
for i in tfidf_matrix.toarray():
    print zip(terms_name, i)

[(u'and', 0.0), (u'bat', 0.0), (u'cat', 0.78800079617844954), (u'fat', 0.0), (u'mat', 0.23270298212286766), (u'on', 0.23270298212286766), (u'rat', 0.0), (u'sat', 0.23270298212286766), (u'the', 0.46540596424573533)]
[(u'and', 0.0), (u'bat', 0.0), (u'cat', 0.0), (u'fat', 0.57989687146162439), (u'mat', 0.34249643393071422), (u'on', 0.34249643393071422), (u'rat', 0.44102651785124652), (u'sat', 0.34249643393071422), (u'the', 0.34249643393071422)]
[(u'and', 0.50165133177159349), (u'bat', 0.50165133177159349), (u'cat', 0.0), (u'fat', 0.0), (u'mat', 0.29628335772067432), (u'on', 0.29628335772067432), (u'rat', 0.38151876810273028), (u'sat', 0.29628335772067432), (u'the', 0.29628335772067432)]

As you can see in the results, the same words will have different scores in each document, because tf-idf is a score of that term within each document. So these are the two methods available to you, so depending on what you want, you can choose what's better for your purposes.