在 word tokenizer 中出现错误“AttributeError: 'numpy.ndarray' object has no attribute 'lower'”

Question

I am trying to train a model to classify multi-label data set by referring this article. I am entirely new to this field and I am getting this error "AttributeError: 'numpy.ndarray' object has no attribute 'lower'"

Here is my code

reviews = pd.read_csv("/content/drive/My Drive/Interim Project Data/score.csv")

Review_labels = reviews[["natral score", "man-made score", "sport score", "special event score"]]
Review_labels.head()

def preprocess_text(sen):
    # Remove punctuations and numbers
    sentence = re.sub('[^a-zA-Z]', ' ', sen)
    sentence = re.sub(r"\s+[a-zA-Z]\s+", ' ', sentence)
    sentence = re.sub(r'\s+', ' ', sentence)
    return sentence

X = []
sentences = list(reviews["User Review"])
for sen in sentences:
    X.append(preprocess_text(sen))

y = Review_labels.values

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=123)
print(len(X_train))
print(len(X_test))

from numpy import array
from keras.preprocessing.sequence import pad_sequences
from keras.preprocessing.text import Tokenizer


tokenizer = Tokenizer(num_words=5000)
tokenizer.fit_on_texts(X)

X_train = tokenizer.texts_to_sequences(X_train)
X_test = tokenizer.texts_to_sequences(X_test)

The error is here

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-128-ff374d8c5eb4> in <module>()
      7 tokenizer.fit_on_texts(X)
      8 
----> 9 X_train = tokenizer.texts_to_sequences(X_train)
     10 X_test = tokenizer.texts_to_sequences(X_test)
     11 

2 frames
/usr/local/lib/python3.6/dist-packages/keras_preprocessing/text.py in text_to_word_sequence(text, filters, lower, split)
     41     """
     42     if lower:
---> 43         text = text.lower()
     44 
     45     if sys.version_info < (3,):

AttributeError: 'numpy.ndarray' object has no attribute 'lower'

here is the data set that I used in the code

It will be really helpful if someone can help with this issue.

Answer 1

Is text a str variable? If it's not maybe you could do

text = str(text).lower()

as long as it's value is something that can be turned into a string

Answer 2

How about trying

tokenizer = Tokenizer(num_words=5000, lower = False)

instead of

tokenizer = Tokenizer(num_words=5000)