Running RNN in Tensorflow

Question

If I have an array of 20 elements of type float.
Based on the values of the first ten elements I want a RNN to predict what the value of the last ten elements are. Using various online resources and books I have gotten a RNN built that reads the first 10 elements and processes them. However I don't know how to get it to use the last ten elements as an 'answer key' and train based off that.

# To support both python 2 and python 3
from __future__ import division, print_function, unicode_literals

# Common imports
import numpy as np
import os
import tensorflow as tf
import numpy as np
import pymysql as pym

# to make this notebook's output stable across runs
def reset_graph(seed=42):
    tf.reset_default_graph()
    tf.set_random_seed(seed)
    np.random.seed(seed)

conn = pym.connect("host.docker.internal","root","","DynaSystems" )
cursor = conn.cursor()
cursor.execute("USE DynaSystems")
cursor.execute("SELECT * FROM simulation")
D = []
for row in cursor:
    D.append(np.fromiter(row, dtype=float, count=-1))
#print(D)

cursor.close()
conn.close()

#get data into a np array
data_np = np.asarray(D, np.float32)
steps = data_np[0:,2:12]
steps = steps.tolist()

a = []
for x in steps:
    c = []
    c.append(x)
    a.append(c)
#get evars out of simulation data
#print(a)

#Rough draft running a Dynamic unrolling and a Basic RNN Cell.
#It works but there's not training and thus no learning happening yet...

n_steps = 1
n_inputs = 10
n_neurons = 10

reset_graph()

X = tf.placeholder(tf.float32, [None, n_steps, n_inputs])

basic_cell = tf.contrib.rnn.BasicRNNCell(num_units=n_neurons)
outputs, states = tf.nn.dynamic_rnn(basic_cell, X, dtype=tf.float32)

init = tf.global_variables_initializer()

with tf.Session() as sess:
    init.run()
    outputs_val = outputs.eval(feed_dict={X: a})

print(outputs_val)

The data in "a" which I'm giving to the feed dict looks something like this:

[[[0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]], [[0.800000011920929, 0.5, 0.800000011920929, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]], [[0.800000011920929, 0.5, 0.800000011920929, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]], [[0.800000011920929, 0.5, 0.800000011920929, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]], [[0.800000011920929, 0.5, 0.800000011920929, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]], [[0.800000011920929, 0.5, 0.800000011920929, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]], [[0.800000011920929, 0.5, 0.800000011920929, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]], [[0.800000011920929, 0.5, 0.800000011920929, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]], [[0.800000011920929, 0.5, 0.800000011920929, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]], [[0.800000011920929, 0.5, 0.800000011920929, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]], [[0.800000011920929, 0.5, 0.800000011920929, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]], [[0.800000011920929, 0.5, 0.800000011920929, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]], [[0.800000011920929, 0.5, 0.800000011920929, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]], [[0.800000011920929, 0.5, 0.800000011920929, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]], [[0.800000011920929, 0.5, 0.800000011920929, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]]]

in the step where I sliced the data_np like so : steps = data_np[0:,2:12]

I got those first ten numbers successfully but how do I grab the last ten and feed them in so as to train the network? I'm assuming the end of my code needs to look something like below, where the y placeholder holds the 'answer key' for the RNN. However, I cannot make it come together.

n_steps = 1
n_inputs = 10
n_neurons = 10
n_outputs = 10
learning_rate = 0.001

reset_graph()

X = tf.placeholder(tf.float32, [None, n_steps, n_inputs])
y = tf.placeholder(tf.int32, [None])

basic_cell = tf.contrib.rnn.BasicRNNCell(num_units=n_neurons)
outputs, states = tf.nn.dynamic_rnn(basic_cell, X, dtype=tf.float32)
logits = tf.layers.dense(states, n_outputs)
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)

loss = tf.reduce_mean(xentropy)
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
training_op = optimizer.minimize(loss)
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))

init = tf.global_variables_initializer()

with tf.Session() as sess:
    init.run()
    outputs_val = outputs.eval(feed_dict={X: a})

print(outputs_val)

Answer 1

First off, have a look into Keras - it's a module which uses TensorFlow as its back-end, but wraps the most important neural net bits in very easy to use objects. The RNN documentation can be found here .

So what I understand from the question is that you have a sequence of numbers and you want to use the previous ones to predict future ones. If each data point you have represents a time step within that sequence you can go one of two ways I believe. This depends on what it is you are trying to model. Have a read in this article which gives a better understanding of LSTM networks then come back here. The two ways are:

1. Many to one data relationship

If your data is just a sequence of steps following one another you can define each time step to be the output of the previous time steps. This means that at t[0] the expected output is t 1 . To put model this you need to put the data into a numpy array with the following shape:

input shape: (number of samples, number of time steps, input data)
i.e. (1, 1, 1) would mean you have 1 sample with 1 step and 1 feature dimension

output shape: (number of samples, output data)
i.e. (1, 1) would mean you have 1 sample with 1 output dimension

And to directly translate this to your example:

The shape could be something like this: (20, 1, 1) where you have 20 samples each with 1 step and 1 feature dimension. Then input your numpy array would look something like [ [[0.5]], [[0.5]], [[0.5]] ... 20 times ] and your output array would be [[0.5], [0.5], [0.5] ... 20 times]

By doing this your neural net will feed in the input 1 step at a time and use all previous steps to predict the next one. For example if you want to predict the 11th step in the sequence of 20, your neural net will use the previous 10 steps to do so. You can think of this a t[0-10] => t[11]

2. Many to many relationship

If you really need to preserve the relationship you described in your question - first 10 steps predicting the remaining ten - you need to use a many to many relationship. Karpathy's article touches on this topic so have a look there. To be honest, I haven't got much experience with such a situation so the only thing I can point at is that you need to use Keras' TimeDistributed Dense layer in order to model this.

I hope this helps. Good luck!