为线性回归预处理文本是否有正确的步骤？

Question

i've combined two different datasets so that one column has text and another column has the sentiment score (binary 0, 1)

I'm trying to make a linear regression model that predicts sentiment based on words used in the text, so far to preprocess the text, i changed the text to lowercase for all texts.

i'm wondering what the next step is after this? i've read up a bit but i'm thinking i may not have the steps in the correct order.

1. lowercase                         1. lowercase
2. remove punctuation               2. tokenize
3.tokenize                          3. remove punctuation

which way is more correct, if i remove the punctuation first i might lose details such as don't and can't.

Answer 1

Preprocessing text for machine learning is usually involved two steps: (I) clearning text, and (II) transforming text to number (aka. embedding). Selecting techniques for these two steps is quite depending on the task and they are related to eachother.

(I) Clearning text : usually involves to (i) handling the case of text, (ii) handling punctuations, (iii) handling stopwords

(i) Handling the case of text : if your text is an English corpus and the selected embedding technique is for similarity measure related task, then it's better to convert all the text/corpus to lowercase. However, if your tasks (eg., tagging, machine translation, etc.) using word embeddings as input representation of words in a sequence model then text casing might matter. It's better to convert the text to lower case before embedding for your regression task.

(ii) Handling punctuations ,"#$%&\'()*+.-:/;?<=>?@[\\]^_ {|}~`: if you use word embedding techniques for similarity related tasks, then you can clean/eliminate punctuations with substituations (eg, replace with ' ') from your text corpus. The word embedding for those tasks can be Bag of Words (BoW), Word2Vect, etc. For your specific task here (regression), then it's good to clean punctuations with substitution of ' ' . For some applications (eg, multilangual machine translation), punctuation might be important.

(iii) handling stopwords : Stop word (eg, the, i, he, …) is a word appeared with a very high frequency on a corpus. Stopwords usually don't provide useful information for the context or the true meaning of a sentence. Common NLP library such as NTK, gensim, spaCy, sklearn provided list of stopwords for some languages. For the similarity related tasks, it's better to remove stopwords before doing embeddings. Removing stopwords is applied for your task (regression). Stopwords can be useful and should not be removed before learning embedding in some other tasks (eg, machine translations). It's better to remove stopwords for your regression task

(II) Transforming text to number (embedding): To be able to fit text data to a machine learning model (eg, your regression model), you need to transform the text data to vectors of number. Tokenization is required before this transformation process. In NLP/ML, this transformation process is called embedding . There are many different approaches to do Word Embeddings in NLP (eg, frequency term (BoW), co-occorrence statistics (GloVe), probabilistic model (LDA2Vec), neural networks (Word2Vec, FastText, BIRD, ...) based approaches). Each technique has its pros and cons. Selecting a word embedding technique is very depending on your application/task. There will be not enough space to write about each word embedding approach/technique here.

The followings are some online tutorials for working with text that can help you quick go through and apply to your problem:

1) Sklearn - working with text data ( https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html )

2) NLTK tutorial ( https://www.nltk.org/book/ch01.html )

3) Spacy - Language processing pipelines tutorial ( https://spacy.io/usage/processing-pipelines )

4) How to Clean Text for Machine Learning with Python ( https://machinelearningmastery.com/clean-text-machine-learning-python/ )