Which checkpoint should I select for continue for Object detection training

Question

I start to training until ckpt-7 then I stopped training. Then again I started training but befor I changed pipline config in fine tune chekpoint on my model. I wrote latest check point and I changed its directory. My loss function approximetly 0.899 before stopped to the training. When I continue to train but its start to steps 100 and my loss fuction 15.009.

How can I contiune the model before stopped? What should I do?

I am using centernet model with Colab. Please explain I am new on that topic.

Answer 1

I could understand your question that you could not resume the training where it stopped.

Actually with the updates in TF2, we need not change the finetune checkpoint parameter in the pipeline.config. Re-run the same training script pointing to the same model_dir where your checkpoints are stored.

TF2 will automatically understand and resume from where the training stopped with the help of checkpoints created in the model_dir.