Understanding Tensorflow initialize_all_variables

Question

Why the following code prints 4 different numbers?

# Random seed for reproducibility
tf.set_random_seed(1234)
# Random variable
foo = tf.Variable(tf.random_uniform(shape=(1,1)),name = 'foo')
# Operation to initialize variables
init = tf.initialize_all_variables()
# Run Operations in session
with tf.Session() as sess:
    # Loop 
    for i in range(1,5):
        # Initialize variables
        sess.run(init)
        # Print foo value
        print(sess.run(foo))

I was expecting it to print the same random value 4 times since I am running the initializer at the start of each of the four iterations.

Answer 1

The function tf.set_random_seed() ensures reproducibility. Every time you execute the program, it generates the same sequence.

Example

# Run1
[[0.96046877]]
[[0.8362156]]
[[0.510509]]
[[0.7130234]]

# Run2
[[0.96046877]]
[[0.8362156]]
[[0.510509]]
[[0.7130234]]

This makes sure your code is reproducible. Tensorflow Documentation

Answer 2

I think when you're running sess.run(init) , it initialises a new random number for the session.

The function initialize_all_variables() is just initialising all the numbers as a whole, but sess.run(init) is the code where that session gets its own number.

I'm not sure though. I'm unable to replicate the code to get a better understanding.

Answer 3

Ok, after reading the documentation and @skillsmuggler answer, I realized that tf.set_random_seed(1234) provides reproducibility across sessions . I will try to explain it with an example:

If you run variables initializer sess1.run(init) multiple times in the same session, it will create a random number every time you run it, since each time it is initialized, it creates a random number.

# Run Operations in session
with tf.Session() as sess1:
# Loop 
for i in range(1,5):
    # Initialize variables
    sess1.run(init)
    # Print foo value
    print(sess1.run(foo))

[[0.7720382]]
[[0.8953308]]
[[0.22609258]]
[[0.07619083]]

The random seed ensures that if we run the previous code on a different session, the numbers generated will be the same (reproducibility).

# Run Operations in session
with tf.Session() as sess2:
# Loop 
for i in range(1,5):
    # Initialize variables
    sess2.run(init)
    # Print foo value
    print(sess2.run(foo))

[[0.7720382]]
[[0.8953308]]
[[0.22609258]]
[[0.07619083]]

So, it indeed, ensures reproducibility.