使用Keras进行整数序列预测

Question

I'm trying to code a RNN model that will predict the next number in the integer series. The model loss gets smaller with each epoch, but the predictions never get quite accurate. I've tried many train set sizes and numbers of epochs, but my predicted value is always off from the expected by few digits. Can you give me some hints what to improve or what I'm doing wrong? This is the code:

from keras.models import Sequential
from keras.layers import Dense, Dropout, LSTM
from keras.callbacks import ModelCheckpoint
from keras.utils import np_utils
from keras import metrics
import numpy as np

training_length = 10000
rnn_size = 512
hm_epochs = 30

def generate_sequence(length=10):
    step = np.random.randint(0,50)
    first_element = np.random.randint(0,10)
    first_element = 0
    l_ist = [(first_element + (step*i)) for i in range(length)]
    return l_ist

training_set = []

for _ in range(training_length):
    training_set.append(generate_sequence(10))

feature_set = [i[:-1] for i in training_set]

label_set = [i[-1:] for i in training_set]

X = np.reshape(feature_set,(training_length, 9, 1))
y = np.array(label_set)


model = Sequential()
model.add(LSTM(rnn_size, input_shape = (X.shape[1], X.shape[2]), return_sequences = True))
model.add(Dropout(0.2))
model.add(LSTM(rnn_size))
model.add(Dropout(0.2))
model.add(Dense(y.shape[1], activation='linear'))
model.compile(loss='mse', optimizer='rmsprop', metrics=['accuracy'])

filepath="checkpoint_folder/weights-improvement.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='loss', verbose=1, save_best_only=True, mode='min')
callbacks_list = [checkpoint]

model.fit(X,y,epochs=hm_epochs, callbacks=callbacks_list)

Effects:

After 30 epochs (Loss: 66.39):

1 Sequence: [0, 20, 40, 60, 80, 100, 120, 140, 160]Expected: [180] || Got: [181.86118]

2 Sequence: [0, 11, 22, 33, 44, 55, 66, 77, 88]Expected: [99] || Got: [102.17369]

3 Sequence: [0, 47, 94, 141, 188, 235, 282, 329, 376]Expected: [423] || Got: [419.1763]

4 Sequence: [0, 47, 94, 141, 188, 235, 282, 329, 376]Expected: [423] || Got: [419.1763]

5 Sequence: [0, 4, 8, 12, 16, 20, 24, 28, 32]Expected: [36] || Got: [37.506496]

6 Sequence: [0, 48, 96, 144, 192, 240, 288, 336, 384]Expected: [432] || Got: [425.0569]

7 Sequence: [0, 28, 56, 84, 112, 140, 168, 196, 224]Expected: [252] || Got: [253.60233]

8 Sequence: [0, 18, 36, 54, 72, 90, 108, 126, 144]Expected: [162] || Got: [163.538]

9 Sequence: [0, 19, 38, 57, 76, 95, 114, 133, 152]Expected: [171] || Got: [173.77933]

10 Sequence: [0, 1, 2, 3, 4, 5, 6, 7, 8]Expected: [9] || Got: [9.577981]

...

After 100 epochs (Loss: 54.81):

1 Sequence: [0, 20, 40, 60, 80, 100, 120, 140, 160] Expected: [180] || Got: [181.03535]

2 Sequence: [0, 11, 22, 33, 44, 55, 66, 77, 88] Expected: [99] || Got: [99.15022]

3 Sequence: [0, 47, 94, 141, 188, 235, 282, 329, 376] Expected: [423] || Got: [423.7969]

4 Sequence: [0, 47, 94, 141, 188, 235, 282, 329, 376] Expected: [423] || Got: [423.7969]

5 Sequence: [0, 4, 8, 12, 16, 20, 24, 28, 32] Expected: [36] || Got: [34.47298]

6 Sequence: [0, 48, 96, 144, 192, 240, 288, 336, 384] Expected: [432] || Got: [432.73163]

7 Sequence: [0, 28, 56, 84, 112, 140, 168, 196, 224] Expected: [252] || Got: [251.55792]

8 Sequence: [0, 18, 36, 54, 72, 90, 108, 126, 144] Expected: [162] || Got: [164.81227]

9 Sequence: [0, 19, 38, 57, 76, 95, 114, 133, 152] Expected: [171] || Got: [172.6425]

10 Sequence: [0, 1, 2, 3, 4, 5, 6, 7, 8] Expected: [9] || Got: [8.837313]

Answer 1

Did you try it with a longer sequence? LSTM is not required, because the dependency is not very long. You can give a try with another variant of RNN.

Answer 2

Regarding your examples your input sequence is just (x 2x 3x) etc. This is not a problem for recurrent neural networks. You want to learn a calculation policy, not long term dependecies in features. RNNs are very powerful and can find very complex pattern, but for this they are not the right tool.

To solve such a problem you can take a look at evolutionary algorithms.