Python如何在csv文件中的推文中应用单词袋

Question

I am currently working twitter data analysis and have been working on applying bag of words technique in Python and have been having no luck. Currently I have been able to stream data to be stored in a database with some preprocessing which I then export the tweets into a csv file but stumbling on the next part to use bag of words in order to do machine learning.

I've tried following https://www.kaggle.com/c/word2vec-nlp-tutorial#part-1-for-beginners-bag-of-words however I have had no success and haven't been able to grasp an understanding how how to approach by just looking at either scikit or nltk documentation. Can anyone advise tutorials I can follow to achieve bag of words with Python 3? Thanks for the help

Answer 1

So scikit-learn's CountVectoriser is a good place to start. You'll want to create a vocabulary of some fixed size (N unique words collected from your tweets) so that you can represent each tweet as a fixed length vector, where each position in the vector represents a particular word from your vocabulary, and the value is the number of times that word has appeared.

With pure Python this would be:

1. Create an array of tweet texts
2. Initialise an empty set representing your vocabulary

First pass through tweets

3. For each tweet, extract unique words
   • Add these words to your vocabulary if they don't exist

Second pass through same tweets

4. For each tweet, extract unique words
   • Create a vector filled with zeros of size N representing the tweet
   • For each word, increment the count corresponding to the position of the word in the vector

You could use 1 or 0 for a word being present or not instead of word frequencies. See what works.

However, scikit-learn makes all of this much easier to do.

I found this tutorial which might help too.