Tensorflow 自定义 Object 检测器：model_main_tf2 未开始训练

Question

Problem summary : The tensorflow custom object detector never starts fine-tuning when i follow the guide in docs. It doesn't throw an exception either.

What i've done: I have installed the object detector api and run a succesful test as according to the docs .

I then followed the guide about training a custom object detector algorithm here , including modifying the pipeline.config file. As per the guide i run

model_main_tf2.py  --model_dir=<path1> --pipeline_config_path=<path2> --alsologtostderr

where path1 and path2 are paths like

 D:/COCO/models/workspace/duck-demo/pre-trained-models/efficientdet_d1_coco17_tpu-32/pipeline.config

The output is shown below. The output, including its many warnings, is expected output as per the guide. However, it was expected to start training afterwards. Instead it just returns, without error nor training. What seems to be the problem here?

output:

...
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor._bifpn_stage.node_input_blocks.7.0.1.1.gamma
W0326 09:24:46.180965 16300 util.py:160] Unresolved object in checkpoint: (root).model._feature_extractor._bifpn_stage.node_input_blocks.7.0.1.1.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor._bifpn_stage.node_input_blocks.7.0.1.1.beta
W0326 09:24:46.180965 16300 util.py:160] Unresolved object in checkpoint: (root).model._feature_extractor._bifpn_stage.node_input_blocks.7.0.1.1.beta
...
WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. 
Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.
W0326 09:24:46.181965 16300 util.py:168] A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. 
Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.

Answer 1

There is a GitHub issue here with many possible solutions being discussed for different types of TensorFlow 2 models for your problem. There's a good chance one of them would help.

Just as a rule of thumb, it's a good idea to always test your installation by running the command python object_detection/builders/model_builder_tf2_test.py before actually proceeding to train anything to diagnose any possible issues early

Answer 2

I had a similar issue, just posting here because after many agonizing hours I finally resolved it, so I wanted to post my fix in case it helps somewhere else.

After spending too much time thinking the issue was with my model variables, I went back and went over every step in the TF object detection api installation to make sure nothing was wrong. I then found that my Windows 10 environmental variables was pointed at the CUDA 11.2 version but I had 11.3 installed, I changed the path and it worked perfectly. I would recommend anyone who isn't getting an error message to check they installed the environment correctly.

Answer 3

Just wait, it can take a while and this is something the developers warned about :

The output will normally look like it has “frozen”, but DO NOT rush to cancel the process. The training outputs logs only every 100 steps by default, therefore if you wait for a while, you should see a log for the loss at step 100.

The time you should wait can vary greatly, depending on whether you are using a GPU and the chosen value for batch_size in the config file, so be patient.

If it's not crashing, it seems like it's working. There's a logging parameter you can change in model_main_tf2.py somewhere. You can decrease from 100 to like 5 or 10 if you want to see verbose more frequently.