Python语言包

Question

[PYTHON 3.x] Hello everyone, I am working on a project in Natural language processing and need some help. I have created a vocabulary (list) of distinct words from all the documents. I want to create a vector of each document against this vocabulary list. (Doc_POS_words contains 100 documents, in this form Doc_POS_words[0] = 1st doc,Doc_POS_words[1] = 2nd doc and so on.)

Output:

# Doc_POS_words = [contains all the words of each document as below]

Doc_POS_words = [
  ['war','life','travel','live','night'], 
  ['books','stuent','travel','study','yellow'],
  ]

# myVoc = [distinct words from all the documents as below]

myVoc = [
  'war',
  'life', 
  'travel',
  'live',
  'night',
  'books',
  'student',
  'study',
  'yellow'
]

# myVoc_vector = [ need this as well ]

# Doc_POS_words_BoW = [need this for each document]

PS: I am not using NLTK because I am not working on any of the supported languages by NLTK

Thanks.

Answer 1

Check TfidfVectorizer

from sklearn.feature_extraction.text import TfidfVectorizer
corpus = ["Doc 1 words",
          "Doc 2 words"]
vectorizer = TfidfVectorizer(min_df=1)
vectors = vectorizer.fit_transform(corpus)

Answer 2

I'm still not sure what you are asking so I will give you some general help. I think what you need is to use python sets.

https://docs.python.org/3/tutorial/datastructures.html#sets

Here are some examples for you, using the data in your question:

# create a set of the whole word list
myVocSet = set(myVoc)

for doc_words in Doc_POS_words:
  # convert from list to set
  doc_words = set(doc_words)  

  # want to find words in the doc also in the vocabulary list?
  print(myVocSet.intersection(doc_words))

  # want to find words in your doc not in the vocabulary list?
  print(doc_words.difference(myVocSet))

  # want to find words in the vocab list not used in your doc?
  print(MyVocSet.difference(myVocSet))

---

Here is some more to help:

>>> x = set(('a', 'b', 'c', 'd'))
>>> y = set(('c', 'd', 'e', 'f'))
>>>
>>> x.difference(y)
{'a', 'b'}
>>> y.difference(x)
{'f', 'e'}
>>> x.intersection(y)
{'c', 'd'}
>>> y.intersection(x)
{'c', 'd'}
>>> x.union(y)
{'a', 'b', 'd', 'f', 'e', 'c'}
>>> x.symmetric_difference(y)
{'a', 'b', 'f', 'e'}