Estimator API: AttributeError: 'NoneType' object has no attribute 'dtype'

Question

I have already looked up the previous answers to this problem but it has not been resolved yet. I am implementing a YOLO algorithm (for object detection) from scratch and am having problem in training part.

For training, I am tf.estimator API and am using a code similar to CNN MNIST code in tensorflow example . I am getting the following error:

Traceback (most recent call last):
  File "recover_v3.py", line 663, in <module>
    model.train(input_fn=train_input_fn, steps=1)
  File "/home/nyu-mmvc-019/miniconda3/envs/tf_0/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 376, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/home/nyu-mmvc-019/miniconda3/envs/tf_0/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1145, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/home/nyu-mmvc-019/miniconda3/envs/tf_0/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1170, in _train_model_default
    features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
  File "/home/nyu-mmvc-019/miniconda3/envs/tf_0/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1133, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "recover_v3.py", line 584, in cnn_model_fn
    loss=loss, global_step=tf.train.get_global_step())
  File "/home/nyu-mmvc-019/miniconda3/envs/tf_0/lib/python3.6/site-packages/tensorflow/python/training/optimizer.py", line 400, in minimize
    grad_loss=grad_loss)
  File "/home/nyu-mmvc-019/miniconda3/envs/tf_0/lib/python3.6/site-packages/tensorflow/python/training/optimizer.py", line 494, in compute_gradients
    self._assert_valid_dtypes([loss])
  File "/home/nyu-mmvc-019/miniconda3/envs/tf_0/lib/python3.6/site-packages/tensorflow/python/training/optimizer.py", line 872, in _assert_valid_dtypes
    dtype = t.dtype.base_dtype
AttributeError: 'NoneType' object has no attribute 'dtype'

The code related to loss function in the main file is as shown(similar to official CNN MNIST example):

if mode == tf.estimator.ModeKeys.TRAIN:
    # This gives the LOSS for each image in the batch.
    # It is importing loss function from another file (called loss_fn)
    # Apparently it returns None (not sure)
    loss = loss_fn.loss_fn(logits, labels)

    optimizer = tf.train.AdamOptimizer(learning_rate=params["learning_rate"])

    train_op = optimizer.minimize(
        loss=loss, global_step=tf.train.get_global_step())

    # Wrap all of this in an EstimatorSpec.
    spec = tf.estimator.EstimatorSpec(
        mode=mode,
        loss=loss,
        train_op=train_op,
        eval_metric_ops=None)

    return spec

Previous answers to similar problem suggested that the loss function is returning nothing. However, when I try the loss function with randomly generated arrays, it works fine and yields normal values.

Also, if I return a constant like 10.0 from loss function, I still get the same error.

I am not sure how to proceed now. Also, is there any way I could print the loss returned by the loss function. Apparently, tf.estimator API start a tensorflow session by itself, and if I try to create another session (in order to print the value returned by loss function), I get other errors.

Answer 1

However, when I try the loss function with randomly generated arrays, it works fine and yields normal values.

It seems that you have an issue with your input_fn. Are you sure that it is correctly implemented ?

Also, is there any way I could print the loss returned by the loss function.

Estimator prints the value of the loss function automatically in the console every global_step % 'save_summary_steps'. You can also track the loss function using a scalar summary like so :

tf.summary.scalar('loss', loss)