如何将此数据集拆分为训练集和验证集？

Question

I want to split dataset with sklearn because I don't think validation_split is working for me. Here's how im actually reading dataset:

input_sentences = []
output_sentences = []
output_sentences_inputs = []    #Translated data

count = 0
for line in open(r'/content/drive/My Drive/TEMPPP/123.txt', encoding="utf-8"):
    count += 1

    if count > NUM_SENTENCES:
        break

    if '\t' not in line:
        continue

    input_sentence, output = line.rstrip().split('\t')

    output_sentence = output + ' <eos>'
    output_sentence_input = '<sos> ' + output

    input_sentences.append(input_sentence)
    output_sentences.append(output_sentence)
    output_sentences_inputs.append(output_sentence_input)

Now im confused how to use scikit learn. For now this is what I did.

from sklearn.model_selection import train_test_split
xTrain, xTest, yTrain, yTest = train_test_split(input_sentences, output_sentences, test_size = 0.2, random_state = 1)

First thing, Is it the right approach????

If no? then how do I make the split?

If yes? then help me with this confusion: I was passing input_sentences and output_sentences to my layers, now what do I need to pass???? Do I still pass input_sentences and output_sentences like before and train model with full dataset or do I need to only send xTrain and yTrain???? And xTest and yTest will never be passed by layers, only used to validate?

Answer 1

Based on your code it seems that what you're currently doing is correct.

No, you can forget about input_sentences and output_sentences , from now only use arrays that train_test_split creates.

If you're using ML algorithm that has a fit() method - this one will use xTrain and yTrain . The predict() method will take xTest and you use yTest to check accuracy of your predictions from predict() method. Say, calling sklearn.metrics.r2_score(predictions, yTest) .

Also please note that using multiple question marks to end your sentence makes it sound interrogatory and impolite . You're here asking for help so please watch your punctuation.