How can I make sentiment analysis with new sentence on trained model?

Question

I trained a model by using Naive Bayes. I have high accuracy, but now I want to give a sentence then I want to see it's sentiment. Here it is my code:

# data Analysis
import pandas as pd

# data Preprocessing and Feature Engineering
from textblob import TextBlob
import re
from nltk.corpus import stopwords
from sklearn.feature_extraction.text import TfidfVectorizer

# Model Selection and Validation
from sklearn.naive_bayes import MultinomialNB
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix, classification_report, accuracy_score
import joblib

import warnings
import mlflow

warnings.filterwarnings("ignore")

train_tweets = pd.read_csv('data/train.csv')

tweets = train_tweets.tweet.values
labels = train_tweets.label.values

processed_features = []

for sentence in range(0, len(tweets)):
    # Remove all the special characters
    processed_feature = re.sub(r'\W', ' ', str(tweets[sentence]))

    # remove all single characters
    processed_feature= re.sub(r'\s+[a-zA-Z]\s+', ' ', processed_feature)

    # Remove single characters from the start
    processed_feature = re.sub(r'\^[a-zA-Z]\s+', ' ', processed_feature)

    # Substituting multiple spaces with single space
    processed_feature = re.sub(r'\s+', ' ', processed_feature, flags=re.I)

    # Removing prefixed 'b'
    processed_feature = re.sub(r'^b\s+', '', processed_feature)

    # Converting to Lowercase
    processed_feature = processed_feature.lower()

    processed_features.append(processed_feature)


vectorizer = TfidfVectorizer(max_features=2500, min_df=7, max_df=0.8, stop_words=stopwords.words('english'))
processed_features = vectorizer.fit_transform(processed_features).toarray()

X_train, X_test, y_train, y_test = train_test_split(processed_features, labels, test_size=0.2, random_state=0)

text_classifier = MultinomialNB()
text_classifier.fit(X_train, y_train)

predictions = text_classifier.predict(X_test)

print(confusion_matrix(y_test,predictions))
print(classification_report(y_test,predictions))
print(accuracy_score(y_test, predictions))


joblib.dump(text_classifier, 'model.pkl')

As you can see, I'm saving my model. Now, I want give an input like this:

new_sentence = "I am very happy today"
model.predict(new_sentence)

And I want see something like this as an output:

sentence = "I am very happy today"
sentiment = Positive

How can I do that?

Answer 1

First, put the preprocessing in a function:

def preproc(tweets):
    processed_features = []

    for sentence in range(0, len(tweets)):
        # Remove all the special characters
        processed_feature = re.sub(r'\W', ' ', str(tweets[sentence]))

        # remove all single characters
        processed_feature= re.sub(r'\s+[a-zA-Z]\s+', ' ', processed_feature)

        # Remove single characters from the start
        processed_feature = re.sub(r'\^[a-zA-Z]\s+', ' ', processed_feature)

        # Substituting multiple spaces with single space
        processed_feature = re.sub(r'\s+', ' ', processed_feature, flags=re.I)

        # Removing prefixed 'b'
        processed_feature = re.sub(r'^b\s+', '', processed_feature)

        # Converting to Lowercase
        processed_feature = processed_feature.lower()

        processed_features.append(processed_feature)

    return processed_features

processed_features = preproc(tweets)
vectorizer = TfidfVectorizer(max_features=2500, min_df=7, max_df=0.8, stop_words=stopwords.words('english'))
processed_features = vectorizer.fit_transform(processed_features).toarray()

Then use it to preprocess the test string and feed it to the classifier using transform :

# feeding two 1-sentence tweets:
test = preproc([["I hate this book."], ["I love this movie."]])
predictions = text_classifier.predict(vectorizer.transform(test).toarray())
print(predictions) 

Now, depending on what labels you have in the dataset and how train_tweets.label.values is coded, you will get different output that you can parse into a string. For example, if the labels in the dataset are coded as 1=positive and 0=negative, you might get [0,1].