tensorflow：您的输入数据用完

Question

I am working on a seq2seq keras/tensorflow 2.0 model. Every time the user inputs something, my model prints the response perfectly fine. However on the last line of each response I get this:

You: WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least steps_per_epoch * epochs batches (in this case, 2 batches). You may need to use the repeat() function when building your dataset.

The "You:" is my last output, before the user is supposed to type something new in. The model works totally fine, but I guess no error is ever good, but I don't quite get this error. It says "interrupting training", however I am not training anything, this program loads an already trained model. I guess this is why the error is not stopping the program?

In case it helps, my model looks like this:

intent_model = keras.Sequential([
    keras.layers.Dense(8, input_shape=[len(train_x[0])]),  # input layer
    keras.layers.Dense(8),  # hidden layer
    keras.layers.Dense(len(train_y[0]), activation="softmax"),  # output layer
])

intent_model.compile(optimizer="adam", loss="categorical_crossentropy", metrics=["accuracy"])
intent_model.fit(train_x, train_y, epochs=epochs)

test_loss, test_acc = intent_model.evaluate(train_x, train_y)
print("Tested Acc:", test_acc)

intent_model.save("models/intent_model.h5")

Answer 1

To make sure that you have " at least steps_per_epoch * epochs batches ", set the steps_per_epoch to

steps_per_epoch = len(X_train)//batch_size

validation_steps = len(X_test)//batch_size # if you have test data

Then, there will be enough data for every epoch. Importantly, keep in mind that by default, batch_size is 32 in model.fit() .

If you're using a tf.data.Dataset , you can also add the repeat() method, but be careful: it will loop indefinitely (unless you specify a number).

Answer 2

I have also had a number of models crash with the same warnings while trying to train them. The training dataset if created using the tf.keras.preprocessing.image_dataset_from_directory() and split 80/20. I have created a variable to try and not run out of image. Using ResNet50 with my own images.....

TRAIN_STEPS_PER_EPOCH = np.ceil((image_count*0.8/BATCH_SIZE)-1)
# to ensure that there are enough images for training bahch
VAL_STEPS_PER_EPOCH = np.ceil((image_count*0.2/BATCH_SIZE)-1)

but it still does. BATCH_SIZE is set to 32 so i am taking 80% of the number of images and dividing by 32 then taking away 1 to have surplus.....or so i thought.

history = model.fit(
        train_ds,
        steps_per_epoch=TRAIN_STEPS_PER_EPOCH,
        epochs=EPOCHS,
        verbose = 1,
        validation_data=val_ds,
        validation_steps=VAL_STEPS_PER_EPOCH,
        callbacks=tensorboard_callback)

Error after 3 hours processing aa single successful Epoch is:

Epoch 1/25 374/374 [==============================] - 8133s 22s/step - loss: 7.0126 - accuracy: 0.0028 - val_loss: 6.8585 - val_accuracy: 0.0000e+00 Epoch 2/25 1/374 [..............................] - ETA: 0s - loss: 6.0445 - accuracy: 0.0000e+00WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least steps_per_epoch * epochs batches (in this case, 9350.0 batches). You may need to use the repeat() function when building your dataset.

this might help....

> > print(train_ds) <BatchDataset shapes: ((None, 224, 224, 3), (None,)), types: (tf.float32, tf.int32)>
> 
> print(val_ds) BatchDataset shapes: ((None, 224, 224, 3), (None,)),types: (tf.float32, tf.int32)>
> 
> print(TRAIN_STEPS_PER_EPOCH)
> 374.0
> 
> print(VAL_STEPS_PER_EPOCH)
> 93.0

Answer 3

Solution which worked for me was to set drop_remainder=True while generating the dataset. This automatically handles any extra data that is left over.

For example:

dataset = tf.data.Dataset.from_tensor_slices((images, targets)) \
        .batch(12, drop_remainder=True)

Answer 4

I had the same problem in TF 2.1. It has something to do with the shape/ type of the input, namely the query. In my case, I solved the problem as follows: (My model takes 3 inputs)

x = [[test_X[0][0]], [test_X[1][0]], [test_X[2][0]]]
MODEL.predict(x)

Output:

WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least steps_per_epoch * epochs batches (in this case, 7 batches). You may need to use the repeat() function when building your dataset.

array([[2.053718]], dtype=float32)

Solution:

x = [np.array([test_X[0][0]]), np.array([test_X[1][0]]), np.array([test_X[2][0]])]
MODEL.predict(x)

Output:

array([[2.053718]], dtype=float32)

Answer 5

我遇到了同样的问题，将validation_steps 从 50 减少到 10 解决了这个问题。

Answer 6

I understand that it is completely fine. Firstly. it is a warning not an error. Secondly, the situation is similar to one data is trained during one epoch, next epoch trains next data and you have set epochs value too high eg500 (assuming your data size is not fixed but will approximately be <=500). Suppose data size is 480. Now, the remaining epoch don't have any data left to process, hence the warning. As a result, it returns to the recent state when the last data was trained. I hope this helps. Do let me know if the concept is misunderstood. Thanks!

Answer 7

Try reducing the steps_per_epoch value below the value you have currently set. This helped me solve the problem

Answer 8

Remove steps_per_epoch and validation_steps parameters in model.fit , if you create a dataset with image_dataset_from_directory .

because the steps was initiated when batch_size passed into image_dataset_from_directory

You can trying get the length of dataset created with image_dataset_from_directory , it is steps number.

Answer 9

我在 google colab 中训练模型时也得到了这个，原因是没有足够的内存/RAM 来存储每批的数据量（如果你使用的是批），所以在我降低 batch_size 之后，它都运行得很好