How to restore a fine-tuned model with Tensorflow 2 Object Detection API for testing?

Question

I have trained (fine-tuned) successfully and validated object detection model from Tensorflow Model Zoo 2, with this config:

... 
train_input_reader: {
  label_map_path: "/label_map.pbtxt"
  tf_record_input_reader {
    input_path: "/train.record"
  }
}

eval_config: {
  metrics_set: "coco_detection_metrics" #coco_detection_metrics
  use_moving_averages: false
  batch_size: 1;
}

eval_input_reader: {
  label_map_path: "/label_map.pbtxt"
  shuffle: false
  num_epochs: 1
  tf_record_input_reader {
    input_path: "/validation.record"
  }
}
...

Then I noticed by analyzing the performance on Tensorboard that the best model based on eval loss is at step 13k ie ckpt-14. However, I also have /test.record on which I want to test the model based on ckpt.14. What could I do? I tried to create a separate folder with ckpt-14.index e ckpt-14.data-... and the file named "checkpoint" containing only ckpt-14 and its timestamp and then launched the evaluation process by replacing validation.record with test.record. in tf_record_input_reader.

It's correct? is there a proper way to testing a model based on a checkpoint with tensorflow 2 object detection api?

Answer 1

You can train and test on the same model simultaneously......But if you have a single GPU, and training with a large dataset, it may not be possible to run testing with the same GPU, as it would result in memory errors.....One good way is to to use the same code and use a work around to do the testing using CPU.......The testing cycle takes place once every 1000 steps and on Tensorboard, you can see both test and eval, and you will also see the bounding boxes with the ground truth side-by-side......

I will try to share the codes for concurrent training and testing.....For training, it will use the GPU, and for testing it will use the CPU.....It has been working for me and no doubt, it should work for you too.....