how to create Keras multi LSTM layer using for loop?

Question

I'm trying to implement a multi layer LSTM in Keras using for loop and this tutorial to be able to optimize the number of layers, which is obviously a hyper-parameter. In the tutorial, the author used skopt for hyper-parameter optimization . I used Functional API to create my model. For simplicity, I changed input_tensor 's shape to arbitrary values. My model is:

from keras.layers.core import Dense
from keras.layers import LSTM, Input
from keras.models import Model

from keras.optimizers import RMSprop
from keras.initializers import glorot_uniform, glorot_normal, RandomUniform


input_tensor = Input(shape=(10, 20))

def create_model(learning_rate, num_lstm_layers, num_lstm_units, activation):
    init = glorot_normal(seed=None)
    init1 = RandomUniform(minval=-0.05, maxval=0.05)

    x = Input(shape=(10, 20))


    for i in range(num_lstm_layers):

        name = 'layer_lstm_{0}'.format(i+1)

        if( (i==0) and (num_lstm_layers==1) ):
            x = LSTM(units=num_lstm_units, dropout=0.2, recurrent_dropout=0.2, 
                    return_sequences=False, kernel_initializer=init, 
                    activation=activation, name=name)(x)
        elif(i != (num_lstm_layers-1) ):
            x = LSTM(units=num_lstm_units, dropout=0.2, recurrent_dropout=0.2,  
                    return_sequences=True, kernel_initializer=init, 
                    activation=activation, name=name)(x)
        else:
            x = LSTM(units=num_lstm_units, dropout=0.2, recurrent_dropout=0.2,  
                    return_sequences=False, kernel_initializer=init, 
                    activation=activation, name=name)(x)
    x = Dense(1, activation='linear', kernel_initializer= init1)(x)

    model = Model(input_tensor, x)

    optimizer = RMSprop(lr=learning_rate, rho=0.9, epsilon=None, decay=0.0)
    model.compile(loss='mean_squared_error', optimizer=optimizer, metrics=['mse'] )

    return model      

Whenever I try to fit the model to the data, I encounter this error :

ValueError: Initializer for variable layer_lstm_1_14/kernel/ is from inside a control-flow construct, such as a loop or conditional. When creating a variable inside a loop or conditional, use a lambda as the initializer.

So far, I've known that somewhere I should add a lambda function or keras Lambda layer . Also, I tested the model in a separate python script, like below:

model = create_model(learning_rate=1e-3, num_lstm_layers=3, num_lstm_units=64, activation='linear')

But again it gives me this error:

ValueError: Graph disconnected: cannot obtain value for tensor Tensor("input_2:0", shape=(?, 10, 20), dtype=float32) at layer "input_2". The following previous layers were accessed without issue: []

I Also tried to create Sequential version of the model. But encountered the same error.

EDIT1: Edited if statement from if( i==0) : to if( (i==0) and (num_lstm_layers==1) ): By doing so and making the changes André suggested, you are able to create LSTM models using for loop.

Answer 1

As I said in the comments, I'm not worried about your for-loop, but rather the input. I'm not 100% sure, but think that you should try to delete

input_tensor = Input(shape=(10, 20)) 

... which comes before the create_model(...) function and edit the creation of the one inside as follows:

input_tensor = x = Input(shape=(10, 20))

Reading on, you say you get Graph disconnected: cannot obtain value for tensor . That definitely sounds like you're input isn't connected. The change I suggest should connect your input and output (respectively the first and second argument of Model(...) ).