无法在Windows上运行Tensorflow对象检测API

Question

I installed tensorflow-gpu 1.4 on my Windows 10 pc. I tried to train a raccoon detector using ssd_mobilenet checkpoint. While I succeeded to this on Google Cloud, it fails to run locally with the following output (apparently it fails to load the pretrained model):

python object_detection/train.py --logtostderr --pipeline_config_path="C://raccoon/training/cfg.config" --train_dir="C://raccoon/training" > out.txt
Instructions for updating:
WARNING:tensorflow:From C:\Users\orona\Dropbox\projects\models\research\object_detection\trainer.py:210: create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Please switch to tf.train.create_global_step
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:Summary name /clone_loss is illegal; using clone_loss instead.
2018-01-22 17:53:34.358048: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2018-01-22 17:53:34.685205: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.683
pciBusID: 0000:01:00.0
totalMemory: 11.00GiB freeMemory: 9.10GiB
2018-01-22 17:53:34.685312: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from C://raccoon/training\model.ckpt
2018-01-22 17:53:41.215719: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_3_3x3_s2_256/weights/ExponentialMovingAverage not found in checkpoint
2018-01-22 17:53:41.216292: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_9_pointwise/weights/RMSProp_1 not found in checkpoint
2018-01-22 17:53:41.217017: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_3_3x3_s2_256/weights/RMSProp not found in checkpoint
2018-01-22 17:53:41.217942: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_3_3x3_s2_256/weights/RMSProp_1 not found in checkpoint
2018-01-22 17:53:41.219617: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_9_pointwise/weights/RMSProp not found in checkpoint
2018-01-22 17:53:41.220835: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_4_3x3_s2_256/BatchNorm/beta/ExponentialMovingAverage not found in checkpoint
2018-01-22 17:53:41.221508: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_4_3x3_s2_256/BatchNorm/beta/RMSProp not found in checkpoint
2018-01-22 17:53:41.223914: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_9_pointwise/weights/ExponentialMovingAverage not found in checkpoint
2018-01-22 17:53:41.223936: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_4_3x3_s2_256/BatchNorm/beta/RMSProp_1 not found in checkpoint
2018-01-22 17:53:41.225563: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_4_3x3_s2_256/BatchNorm/gamma/ExponentialMovingAverage not found in checkpoint
2018-01-22 17:53:41.229052: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_4_3x3_s2_256/BatchNorm/gamma/RMSProp not found in checkpoint
2018-01-22 17:53:41.229776: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_4_3x3_s2_256/BatchNorm/gamma/RMSProp_1 not found in checkpoint
2018-01-22 17:53:41.234741: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_4_3x3_s2_256/weights/ExponentialMovingAverage not found in checkpoint
.
.
.
.
2018-01-22 17:53:41.726224: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_4_pointwise/BatchNorm/gamma/RMSProp_1 not found in checkpoint
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.NotFoundError'>, Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_3_3x3_s2_256/weights/ExponentialMovingAverage not found in checkpoint
         [[Node: save_1/RestoreV2_303 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save_1/Const_0_0, save_1/RestoreV2_303/tensor_names, save_1/RestoreV2_303/shape_and_slices)]]

Caused by op 'save_1/RestoreV2_303', defined at:
  File "object_detection/train.py", line 164, in <module>
    tf.app.run()
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "object_detection/train.py", line 160, in main
    worker_job_name, is_chief, FLAGS.train_dir)
  File "C:\Users\orona\Dropbox\projects\models\research\object_detection\trainer.py", line 317, in train
    keep_checkpoint_every_n_hours=keep_checkpoint_every_n_hours)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1218, in __init__
    self.build()
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1227, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1263, in _build
    build_save=build_save, build_restore=build_restore)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 751, in _build_internal
    restore_sequentially, reshape)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 427, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 267, in restore_op
    [spec.tensor.dtype])[0])
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_io_ops.py", line 1020, in restore_v2
    shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 2956, in create_op
    op_def=op_def)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1470, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

NotFoundError (see above for traceback): Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_3_3x3_s2_256/weights/ExponentialMovingAverage not found in checkpoint
         [[Node: save_1/RestoreV2_303 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save_1/Const_0_0, save_1/RestoreV2_303/tensor_names, save_1/RestoreV2_303/shape_and_slices)]]

Traceback (most recent call last):
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1323, in _do_call
    return fn(*args)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1302, in _run_fn
    status, run_metadata)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 473, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_3_3x3_s2_256/weights/ExponentialMovingAverage not found in checkpoint
         [[Node: save_1/RestoreV2_303 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save_1/Const_0_0, save_1/RestoreV2_303/tensor_names, save_1/RestoreV2_303/shape_and_slices)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "object_detection/train.py", line 164, in <module>
    tf.app.run()
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "object_detection/train.py", line 160, in main
    worker_job_name, is_chief, FLAGS.train_dir)
  File "C:\Users\orona\Dropbox\projects\models\research\object_detection\trainer.py", line 332, in train
    saver=saver)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\contrib\slim\python\slim\learning.py", line 742, in train
    master, start_standard_services=False, config=session_config) as sess:
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\supervisor.py", line 964, in managed_session
    self.stop(close_summary_writer=close_summary_writer)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\supervisor.py", line 792, in stop
    stop_grace_period_secs=self._stop_grace_secs)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\coordinator.py", line 389, in join
    six.reraise(*self._exc_info_to_raise)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\six.py", line 693, in reraise
    raise value
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\supervisor.py", line 953, in managed_session
    start_standard_services=start_standard_services)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\supervisor.py", line 708, in prepare_or_wait_for_session
    init_feed_dict=self._init_feed_dict, init_fn=self._init_fn)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\session_manager.py", line 273, in prepare_session
    config=config)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\session_manager.py", line 205, in _restore_checkpoint
    saver.restore(sess, ckpt.model_checkpoint_path)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1666, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 889, in run
    run_metadata_ptr)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1120, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1317, in _do_run
    options, run_metadata)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1336, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_3_3x3_s2_256/weights/ExponentialMovingAverage not found in checkpoint
         [[Node: save_1/RestoreV2_303 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save_1/Const_0_0, save_1/RestoreV2_303/tensor_names, save_1/RestoreV2_303/shape_and_slices)]]

Caused by op 'save_1/RestoreV2_303', defined at:
  File "object_detection/train.py", line 164, in <module>
    tf.app.run()
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "object_detection/train.py", line 160, in main
    worker_job_name, is_chief, FLAGS.train_dir)
  File "C:\Users\orona\Dropbox\projects\models\research\object_detection\trainer.py", line 317, in train
    keep_checkpoint_every_n_hours=keep_checkpoint_every_n_hours)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1218, in __init__
    self.build()
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1227, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1263, in _build
    build_save=build_save, build_restore=build_restore)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 751, in _build_internal
    restore_sequentially, reshape)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 427, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 267, in restore_op
    [spec.tensor.dtype])[0])
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_io_ops.py", line 1020, in restore_v2
    shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 2956, in create_op
    op_def=op_def)
  File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1470, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

NotFoundError (see above for traceback): Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_3_3x3_s2_256/weights/ExponentialMovingAverage not found in checkpoint
         [[Node: save_1/RestoreV2_303 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save_1/Const_0_0, save_1/RestoreV2_303/tensor_names, save_1/RestoreV2_303/shape_and_slices)]]

Answer 1

Tx David. I found the problem. in windows one should use path with slashes in the pipeline config , but for the command line arguments use the windows backslash.