我将数据拆分为测试集和训练集，错误是“找到样本数量不一致的输入变量：[1000, 23486]”

Question

my project is to classify the reviews as good or bad using nlp. i have imported the data and done the tokenisation, vectorisation using bag of words model. now i have to spilt the data into testing and training sets and i am getting an error saying "Found input variables with inconsistent numbers of samples: [1000, 23486]"

My file has a column called Review Text and i want to classify the reviews as good or bad. i have attached the tsv file that i am using for this project. please do help me in correcting the error and any change in approach that i can do. i have attached the code here too.

My data file here

import numpy as np
import pandas as pd
import nltk
import matplotlib

dataset = pd.read_csv("C:/Users/a/Downloads/data.tsv", delimiter = "\t", quoting = 1)
dataset.head()

import re
import nltk
from nltk.corpus import stopwords
from nltk.stem.porter import PorterStemmer
corpus = []
for i in range(0, 1000):
  review = re.sub('[^a-zA-Z]', ' ', str(dataset['Review Text'][i]))
  review = review.lower()
  review = review.split()
  ps = PorterStemmer()
  review = [ps.stem(word) for word in review if not word in 
  set(stopwords.words('english'))]
  review = ' '.join(review)
  corpus.append(review)

  from sklearn.feature_extraction.text import CountVectorizer
  cv = CountVectorizer(max_features = 1500)
  X = cv.fit_transform(corpus).toarray()
  y = df.iloc[:, 6].values

  from sklearn.model_selection import train_test_split
  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)

Answer 1

Ok, the problem is that X and y must have the same dimensions.

If you want to use just 1000 reviews you can use the same for cycle and then, when selecting y you just do:

y = dataset.iloc[:1000, 6].values

Otherwise, if you want to use the whole dataset you must edit the first part of the cycle.