Getting error with tensorflow tf.control_dependencies and tf.layers.batch_normalization

Question

I'm trying to build a neural net with two loss functions, which are combined like weighed sum. First one simply computes mean square error of linear output of dense layer and given labels, but another makes heavy use of nested tf.map_fn . There are batch norm layers used with tf.layers.batch_normalization() so i had to add these lines to optimization objective:

    with tf.name_scope("Optimizer"):
        with tf.control_dependencies(tf.get_collection(tf.GraphKeys.UPDATE_OPS)):
            adam = tf.train.AdamOptimizer()
            self.train_op = adam.minimize(self.total_loss)

However i get error:

   AttributeError: 'NoneType' object has no attribute 'op'

It comes from minimize() method. If i remove control dependencies, there is no error. Also if i remove second optimization objective which relies on loops, there is no error. I've tested second loss function in forward propagation and it worked just fine.

Any ideas how to track the problem? Full error log:

Traceback (most recent call last):
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3267, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-6d5efdb6d091>", line 1, in <module>
    runfile('/home/mtarasov/PycharmProjects/ML/src/utils/model.py', wdir='/home/mtarasov/PycharmProjects/ML/src/utils')
  File "/home/mtarasov/Installations/pycharm-2018.2.4/helpers/pydev/_pydev_bundle/pydev_umd.py", line 198, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/home/mtarasov/Installations/pycharm-2018.2.4/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/mtarasov/PycharmProjects/ML/src/utils/model.py", line 168, in <module>
    model = Model().build()
  File "/home/mtarasov/PycharmProjects/ML/src/utils/model.py", line 60, in build
    self.train_op = adam.minimize(self.total_loss)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/optimizer.py", line 400, in minimize
    grad_loss=grad_loss)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/optimizer.py", line 514, in compute_gradients
    colocate_gradients_with_ops=colocate_gradients_with_ops)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 596, in gradients
    gate_gradients, aggregation_method, stop_gradients)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 663, in _GradientsHelper
    to_ops, from_ops, colocate_gradients_with_ops, func_graphs, xs)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 190, in _PendingCount
    between_op_list, between_ops, colocate_gradients_with_ops)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1432, in MaybeCreateControlFlowState
    loop_state.AddWhileContext(op, between_op_list, between_ops)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1244, in AddWhileContext
    grad_state = GradLoopState(forward_ctxt, outer_grad_state)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 846, in __init__
    real_cnt, outer_grad_state)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2585, in AddBackpropLoopCounter
    name="b_count")
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 248, in _Enter
    data, frame_name, is_constant, parallel_iterations, name=name)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_control_flow_ops.py", line 178, in enter
    parallel_iterations=parallel_iterations, name=name)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 454, in new_func
    return func(*args, **kwargs)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3155, in create_op
    op_def=op_def)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1746, in __init__
    self._control_flow_post_processing()
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1757, in _control_flow_post_processing
    self._control_flow_context.AddOp(self)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2438, in AddOp
    self._AddOpInternal(op)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2475, in _AddOpInternal
    for x in external_inputs if x.outputs]
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2475, in <listcomp>
    for x in external_inputs if x.outputs]
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 80, in identity
    return gen_array_ops.identity(input, name=name)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3264, in identity
    "Identity", input=input, name=name)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 454, in new_func
    return func(*args, **kwargs)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3155, in create_op
    op_def=op_def)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1746, in __init__
    self._control_flow_post_processing()
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1757, in _control_flow_post_processing
    self._control_flow_context.AddOp(self)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2438, in AddOp
    self._AddOpInternal(op)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2466, in _AddOpInternal
    self._MaybeAddControlDependency(op)
  File "/home/mtarasov/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2504, in _MaybeAddControlDependency
    op._add_control_input(self.GetControlPivot().op)
AttributeError: 'NoneType' object has no attribute 'op

Answer 1

我不知道到底发生了什么，但是我已经将所有层（包括批处理规范）从tf.layers为tf.keras.layers ，错误消失了！

Answer 2

Adding on to mcstarioni's answer. As pointed out, replacing the batch norm layers with tf.keras.layers.BatchNormalization appears to remove the error. However, this is because BatchNormalization in keras does not add the batch norm parameters in UPDATE_OPS as is noted here , as it uses a different way of training. If you check the moving mean and variance, you'll notice they are not updated during training just by running train_op . It is important to run layer.update in addition to train_op , which should solve the problem.

Alternatively, if possible, try to remove the nested map_fn .