初始v3再培训错误（花例子）

Question

I'm currently facing a weird bug with the flower retraining example ( https://www.tensorflow.org/versions/r0.9/how_tos/image_retraining/index.html ).

Tensorflow Release 0.9 was installed from source and I tried to run the image_retraining python script (it does start and create a few bottlenecks but then the following error message appears).

Might anyone have an idea what the problem could be? I didn't find any similar posts to this.

E tensorflow/core/kernels/check_numerics_op.cc:157] abnormal_detected_host @0x10007200300 = {1, 0} activation input is not finite.
Traceback (most recent call last):
  File "examples/image_retraining/retrain.py", line 888, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
    sys.exit(main(sys.argv))
  File "examples/image_retraining/retrain.py", line 798, in main
    jpeg_data_tensor, bottleneck_tensor)
  File "examples/image_retraining/retrain.py", line 456, in cache_bottlenecks
    jpeg_data_tensor, bottleneck_tensor)
  File "examples/image_retraining/retrain.py", line 414, in get_or_create_bottleneck
    bottleneck_tensor)
  File "examples/image_retraining/retrain.py", line 331, in run_bottleneck_on_image
    {image_data_tensor: image_data})
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 382, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 655, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 723, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 743, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.InvalidArgumentError: activation input is not finite. : Tensor had NaN values
         [[Node: conv_1/CheckNumerics = CheckNumerics[T=DT_FLOAT, message="activation input is not finite.", _device="/job:localhost/replica:0/task:0/gpu:0"](conv_1/batchnorm)]]
Caused by op u'conv_1/CheckNumerics', defined at:
  File "examples/image_retraining/retrain.py", line 888, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
    sys.exit(main(sys.argv))
  File "examples/image_retraining/retrain.py", line 769, in main
    create_inception_graph())
  File "examples/image_retraining/retrain.py", line 312, in create_inception_graph
    RESIZED_INPUT_TENSOR_NAME]))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 274, in import_graph_def
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2297, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1231, in __init__
    self._traceback = _extract_stack()

Answer 1

Update: Just to follow up, Tensorflow 1.6 is recommended as many operations are much faster. If you are running an Nvidia GPU, make sure you install CUDA 9.0 and not 9.1, 9.1 will break everything.

For cuDNN, it needs to match both CUDA 9.0 and also the version that Tensorflow was built with. For Tensorflow 1.6, be sure to install version 7.0.4, not 7.1, and the specific version that 1.6 was built with (otherwise, it will also break): The exact version is cuDNN v7.0.4.31-1 for CUDA 9.0 (not 9.1). The latest versions (7.1.2 at this time) will throw errors as Tensorflow 1.6 was built with 7.0.4

Original post: This is a bug in TensorFlow that I have also encountered (I'm using 2x GTX 1080 in Ubuntu 14.04)

One option is to install Cuda 8.0. However, Cuda 8.0 isn't fully supported and you may encounter other issues.

Another way to work around this if you are just experimenting is to build it and run it on CPUs only, at least for the bottleneck generation phase.

bazel build -c opt --copt=-mavx tensorflow/examples/image_retraining:retrain
bazel-bin/tensorflow/examples/image_retraining/retrain --image_dir ~/flower_photos

As you probably know, if you've built TensorFlow with GPU support and then run this:

python tensorflow/examples/image_retraining/retrain.py --image_dir ~/flower_photos

it will run with GPU support and then you'll probably hit the same error.

I've opened an issue here: https://github.com/tensorflow/tensorflow/issues/3560

Until they fix it, the workaround works as long as you don't have a large number of categories to classify for.