InvalidArgumentError when making a stateful LSTM

Question

I'm working on a stateful LSTM to predict stock prices.

These are the shapes of my input data: (updated)

x_train = (10269, 300, 89)
y_train = (10269, 1)
x_test = (4401, 300, 89)
y_test = (4401, 1)

This is my model initialisation:

batch_size = 63
timesteps = x_train.shape[1]
data_dim = x_train.shape[2]

model = Sequential()

model.add(LSTM(32, return_sequences=True, batch_input_shape=(batch_size, timesteps, data_dim), stateful=True))
model.add(LSTM(32, return_sequences=True, stateful=True))
model.add(LSTM(32, stateful=True))

model.add(Dense(1))

model.compile(optimizer = 'adam', loss = 'mean_squared_error')

But when I fit this I get the error:

InvalidArgumentError:    Specified a list with shape [64,89] from a tensor with shape [29,89]
 [[{{node TensorArrayUnstack/TensorListFromTensor}}]]
 [[sequential/lstm/PartitionedCall]] [Op:__inference_train_function_6536]

To my knowledge, I have defined batch_input_shape correctly and don't see what I have done wrong.

Edit:

Some suggested that I try making my sample size divisible by my batch size. I tried that and got the same error.

(I updated my train and test sizes as seen above)

My new batch size is 63 and my data size 10269. 10269/63 = 163. This is the error:

InvalidArgumentError:    Specified a list with shape [63,89] from a tensor with shape [54,89]
 [[{{node TensorArrayUnstack/TensorListFromTensor}}]]
 [[sequential_1/lstm_3/PartitionedCall]] [Op:__inference_test_function_20179]

Answer 1

When using stateful LSTMs your input must be evenly divisible by batch size.

In your case 3697 // 64 is not an integer.

Since 3697 is a prime number, you need to drop a sample make your input size 3696. When you have 3696 samples, change to code according to (model definition stays same):

batch_size = 33 # some number that divides your samples evenly.
timesteps = x_train.shape[1]
data_dim = x_train.shape[2]

model.fit(x_train, y_train, batch_size = batch_size, ...)

Answer 2

This problem has to do with the stateful argument. when it is used, the number of samples should be divisible by the number of samples.

In your case, you have 3697 samples which is no divisible by 64.

So what you can do is remove 49 samples and take 3648 samples only since 3648 is divisible by 64.

The same thing for the number of samples of validation data . you have to change it to a number divisible by the batch size.

Secondly, use: model.fit(x_train, y_train, batch_size=batch_size,validation_data=(x_val,y_val))

If you don't want to remove any samples from your dataset, you can work with a data generator as shown here