带python的词袋模型

Question

I am trying to do a sentimental analysis with python on a bunch of txt documents. I did so far the preprocessing and extracted only the important words from the text, eg I deleted stop-words, the punctuation. Also I created a kind of bag-of-words counting the term frequency. The next step would be to implement a corresponding model.

I am not experienced in machine learning resp. text mining. I am also uncertain about the way I created the bag-of-words model. Could you please have a look at my code and tell me if I am on the right track. I would also like to know if my previous path is a good basis for a model and how do I build on that basis a good model in order to categorize my documents.

This is my code:

import spacy
import string
import os,sys
import re
import numpy as np
np.set_printoptions(threshold=sys.maxsize)
from collections import Counter

# Load English tokenizer, tagger, parser, NER and word vectors
nlp_en = spacy.load("en_core_web_sm")
nlp_de = spacy.load("de_core_news_sm")
path_train = "Sentiment/Train/"
path_test = "Sentiment/Test/"
text_train = []
text_text = []


# Process whole documents
for filename in os.listdir(path_train):
    text = open(os.path.join(path_train, filename),encoding="utf8", errors='ignore').read()
    text = text.replace("\ue004","s").replace("\ue006","y")
    text = re.sub(r'^http?:\/\/.*[\r\n]*', '', text, flags=re.MULTILINE)
    text = "".join(filter(lambda x: x in string.printable, text))
    text = " ".join(text.split())
    text = re.sub('[A-Z]+', lambda m: m.group(0).lower(), text)
    if filename.startswith("de_"):
        text_train.append(nlp_de(text))
    else:
        text_train.append(nlp_en(text))

docsClean = []
for doc in text_train:
    #for token in doc: 
        #print(token.text, token.lemma_, token.pos_, token.tag_, token.dep_,token.shape_, token.is_alpha, token.is_stop)

    cleanWords = [token.lemma_ for token in doc if token.is_stop == False and token.is_punct == False and token.pos_ != "NUM"]
    docsClean.append(cleanWords)

print(docsClean)

for doc in docsClean:

    bag_vector = np.zeros(len(doc))

    for w in doc:
        for i,word in enumerate(doc):
            if word == w:
                bag_vector[i] += 1

    print(bag_vector)

This is how my bow-model looks like:

Answer 1

get_dummies ，您可以尝试使用pandas和get_dummies 。