TF Graph does not correspond to the code

Question

I am trying to create a very simple neural network reading in information with the shape 1x2048 and to create a classification for two categories (object or not object). The graph structure however, deviates from what I believe to have coded. The dense layers should be included in the scope of "inner_layer" and should be receiving their input from the "input" placeholder. Instead, TF seems to be treating them as independent layers which do not receive any information from "input".

Also, when using trying to use tensorboard summaries I get an error telling me that I have not mentioned inserting inputs for the apparent placeholders of the dense layers. When omitting tensorboard, everything works as I expected it based on the code.

I have spent a lot of time trying to find the problem but I think I must be overlooking an something very basic.

The graph I get in tensorboard is on this image .

Which corresponds to the following code:

tf.reset_default_graph()
keep_prob = 0.5

# Graph Strcuture
## Placeholders for input
with tf.name_scope('input'):
    x_ = tf.placeholder(tf.float32, shape = [None, transfer_values_train.shape[1]], name = "input1")
    y_ = tf.placeholder(tf.float32, shape = [None, num_classes], name = "labels")

## Dense Layer one with 2048 nodes
with tf.name_scope('inner_layers'):
    first_layer = tf.layers.dense(x_, units = 2048, activation=tf.nn.relu, name = "first_dense")
    dropout_layer = tf.nn.dropout(first_layer, keep_prob, name = "dropout_layer")
    #readout layer, without softmax
    y_conv = tf.layers.dense(dropout_layer, units = 2, activation=tf.nn.relu, name = "second_dense")

# Evaluation and training
with tf.name_scope('cross_entropy'):
    cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(labels = y_ , logits = y_conv),
                                   name = "cross_entropy_layer")
with tf.name_scope('trainer'):
    train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)

with tf.name_scope('accuracy'):
    prediction = tf.argmax(y_conv, axis = 1)
    correct_prediction = tf.equal(prediction, tf.argmax(y_, axis = 1))
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

Does anyone have an idea why the graph is so different from what you would expect based on the code?

Answer 1

The graph rendering in tensorboard may be a bit confusing (initially), but it's correct. Take a look at this picture where I've left only the inner_layers part of your graph:

You may notice that:

• The first_dense and second_dense are actually the name scopes themselves (generated by tf.layers.dense function; see also this question ).
• Their input/output tensors are inside the inner_layers scope and wire correctly to the dropout_layer . Here, in each of dense layers, live the corresponding linear ops: MatMul , BiasAdd , Relu .

• Both scopes also include the variables ( kernel and bias each), that are shown separately from inner_layers . They encapsulate the ops related specifically to variable, such as read , assign , initialize , etc. The linear ops in first_dense depend on the variable ops of first_dense , and second_dense likewise.

  The reason for this separation is that in distributed settings the variables are manages by a different task called parameter server . It's usually run on a different device (CPU as opposed to GPU), sometimes even on a different machine. In other words, for tensorflow the variable management is by design different from matrix computation.

  Having said that, I'd love to see a mode in tensorflow that would not split the scope into variables and ops and keep them coupled.

Other than this the graph perfectly matches the code.