使 tfidf 向量化器作为文档数量的特征返回

Question

I am using the Sklearn TfidfVectorizer that I fit on N documents and than I want to get a vector representation of a word based on its tfidf score in each documents.

Some code could help:

from sklearn.feature_extraction.text import TfidfVectorizer

model = TfidfVectorizer()

corpus = ["first corpus with words like dog and cat", "second corpus with words like car and plane"]

model.fit(corpus)

model.transform(["cat"]).toarray().shape

>> (1, 11)

Why am I getting 11 feature ? I am expecting to have 2 features as I fitted the model with only two documents.

So What I wont is something like :

[0, tfidfscore]

Answer 1

I read the documentation and with a basic understanding about TF-IDF, I could arrive at a conclusion. This is not an expert opinion.

As per the documentation , transform returns a sparse matrix whose dimensions are (n_samples, n_features).

Returns X:sparse matrix, [n_samples, n_features]

Tf-idf-weighted document-term matrix.

Now your n_samples is 1 and n_features is coming from the model ... which it computed to be equal to 11.

What is returned by transform is TD-IDF weighted Document-term-matrix where every row corresponds to document and every column is features.

You can know your features by saying "print(model.get_feature_names())". In your case it will give the following output.

['and', 'car', 'cat', 'corpus', 'dog', 'first', 'like', 'plane', 'second', 'with', 'words']

As you can see there are 11 features. Now cat is the third element for which the frequency must be high. If you say "print(model.transform(["cat"]).toarray())" you will know the entire matrix. As said before there will be one row (as you passed in one document .. "cat" and 11 columns (due to the reason above). As you can see below at the third column the frequency is highest 1.00.

[[0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]]

I have done minimal changes to your code with an intention that it will help you.

from sklearn.feature_extraction.text import TfidfVectorizer

model = TfidfVectorizer()

corpus = ["first corpus with words like dog and cat", "second corpus with words like car and plane"]

model.fit(corpus)

Returned_Features = model.get_feature_names()
Returned_TF_IDF_DTM = model.transform(["cat"]).toarray()

print(Returned_Features)
print(Returned_TF_IDF_DTM)

I hope it helps. All the best

Answer 2

Seems that you want to do something like this

from sklearn.feature_extraction.text import TfidfVectorizer

model = TfidfVectorizer()

corpus = ["first corpus with words like dog and cat", "second corpus with words like car and plane"]

X = model.fit_transform(corpus)

words = model.get_feature_names()
word2idx = dict(zip(words, range(len(words))))

print(X[:, word2idx['cat']].todense())

This well give you the coressponding raw of the word "cat" in the tf-idf matrix