如何修复错误 ValueError: could not convert string to float in a NLP project in python?

Question

I am writing a python code using jupyter notebook that train and test a dataset in order to return a correct sentiment.

The problem that when i try to predict the sentiment of the phrase the system crash and display the below error:

ValueError: could not convert string to float: 'this book was so interstening it made me not happy'

Note i have an imbalanced dataset so i use SMOTE in order to over_sampling the dataset

code:

import pandas as pd
import numpy as np
from imblearn.over_sampling import SMOTE# for inbalance dataset
from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction.text import TfidfTransformer,TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score,classification_report,confusion_matrix
from sklearn.pipeline import Pipeline

df = pd.read_csv("data/Apple-Twitter-Sentiment-DFE.csv",encoding="ISO-8859-1")

df
# data is cleaned using preprocessing functions

# Solving inbalanced dataset using SMOTE 

vectorizer = TfidfVectorizer()
vect_df =vectorizer.fit_transform(df["clean_text"])
oversample = SMOTE(random_state = 42)
x_smote,y_smote = oversample.fit_resample(vect_df, df["sentiment"])
print("shape x before SMOTE: {}".format(vect_df.shape))
print("shape x after SMOTE: {}".format(x_smote.shape))
print("balance of targets feild %")
y_smote.value_counts(normalize = True)*100


# split the dataset into train and test 
x_train,x_test,y_train,y_test = train_test_split(x_smote,y_smote,test_size = 0.2,random_state =42)


logreg = Pipeline([
                ('tfidf', TfidfTransformer()),
                ('clf', LogisticRegression(n_jobs=1, C=1e5)),
               ])
logreg.fit(x_train, y_train)

y_pred = logreg.predict(x_test)

print('accuracy %s' % accuracy_score(y_pred, y_test))
print(classification_report(y_test, y_pred))

# Make prediction 
exl = "this book was so interstening it made me not happy"

logreg.predict(exl)

Answer 1

You should define your variable exl as the following:

exl = vectorizer.transform(["this book was so interstening it made me not happy"])

and then do the prediction.

First, put the testing data in a list and then use vectorizer to use features extracted from your training data to do the prediction.