custom rnn migration to tensorflow 2.0

Question

I need to migrate the below part in my custom RNN implementation in tensorflow 1.x to tensorflow 2.x.

I am kind of stuck at the place where I need to convert tf.get_variable with initializers such as xaver initializers inside a tf.variable_scope.

I have referred the migration guide, but still I could not understand the conversion of tf.get_variable with xavier initialization, also I have to migrate few placeholders without a pre-defined shape.

        with tf.variable_scope(self._scope):

            with tf.variable_scope("PresentState"):
                self._U = tf.get_variable("U", shape=[self._num_in, self._n_hidden], dtype=tf.float32,
                                             initializer=xavier_initializer())
                self._W = tf.get_variable("W", shape=[self._n_hidden, self._n_hidden],
                                             dtype=tf.float32,
                                             initializer=xavier_initializer())
                self._b = tf.get_variable("B", shape=[self._n_hidden], dtype=tf.float32,
                                            initializer=xavier_initializer())
                self._p = None

Placeholder part.

p = tf.placeholder(tf.float32, shape=[batch_size, None, num_in], name="p")

Answer 1

In idiomatic TF2.x code, you should not use tf.Placeholders, but instead define them as functions arguments to your callable @tf.function . Since TF2.x now supports eager execution, you don't need any tf.Session calls and hence can pass your plain python variables to any @tf.function .

For the first part of your question:

I am not that deep into RNNs, but from your code it looks like you could refactor the inner tf.variable_scope into a tf.Module , which allows you to use xavier initialzation as follows (Not tested):

class PresentState(tf.Module):
  def __init__(self, ins, hiddens, outs):
    initializer = tf.initializers.GlorotNormal() # Xavier initialization
    self._U = tf.Variable(initializer([ins, outs]), dtype=tf.float32)
    self._W = tf.Variable(initializer([ins, outs]), dtype=tf.float32)
    self._b = tf.Variable(initializer([outs]), dtype=tf.float32)
    self._p = None