Tensorflow 2 Object 检测 API - 官方模型：无法更改 params_override 参数中的其他参数

Question

When using any of the Object Detection models from TensorFlow's Official Models in the ModelZoo, there is an argument called params_override. Based on the code here ( https://github.com/tensorflow/models/blob/master/official/modeling/hyperparams/params_dict.py ), it seems like after creating a ParamsDict with a given a set of default parameters, it then overrides certain parameters. The example that they give on the README.md for the RetinaNet model ( https://github.com/tensorflow/models/tree/master/official/vision/detection ) is --params_override="{ type: retinanet, train: { checkpoint: { path: ${RESNET_CHECKPOINT?}, prefix: resnet50/ }, train_file_pattern: ${TRAIN_FILE_PATTERN?} }, eval: { val_json_file: ${VAL_JSON_FILE?}, eval_file_pattern: ${EVAL_FILE_PATTERN?} } }" . These default keys (ie type, train.checkpoint, train.prefix, train.train_file_pattern, eval.val_json_file, eval.eval_file_pattern) works fine. However, when I try to change other parameters, it gives an error, such as: KeyError: 'The key `num_classes:2` does not exist. To extend the existing keys, use `override` with `is_strict` = False.' KeyError: 'The key `num_classes:2` does not exist. To extend the existing keys, use `override` with `is_strict` = False.' . This happens for any of the parameters I try to change beyond what's provided. Other examples include: architecture{...,num_classes:2},eval:{...,eval_samples:100}, train:{...,total_steps:100, batch_size:8} , all of which give the same The key does not exist error. Here is a sample of the params.yaml file that is outputted as part of the default settings.

anchor:
  anchor_size: 4.0
  aspect_ratios: [1.0, 2.0, 0.5]
  num_scales: 3
architecture:
  backbone: resnet
  max_level: 7
  min_level: 3
  multilevel_features: fpn
  num_classes: 91
  parser: retinanet_parser
  use_bfloat16: false
enable_summary: true
eval:
  batch_size: 8
  eval_dataset_type: tfrecord
  eval_file_pattern: annotations\test.record
  eval_samples: 5000
  eval_timeout: null
  input_sharding: true
  min_eval_interval: 180
  num_images_to_visualize: 0
  num_steps_per_eval: 1000
  type: box
  use_json_file: true
  val_json_file: ''
fpn:
  fpn_feat_dims: 256
  use_batch_norm: true
  use_separable_conv: false
isolate_session_state: false
model_dir: probe_detection_models\v1
norm_activation:
  activation: relu
  batch_norm_epsilon: 0.0001
  batch_norm_momentum: 0.997
  batch_norm_trainable: true
  use_sync_bn: false
postprocess:
  max_total_size: 100
  nms_iou_threshold: 0.5
  pre_nms_num_boxes: 5000
  score_threshold: 0.05
  use_batched_nms: false
predict:
  batch_size: 8
resnet:
  resnet_depth: 50
retinanet_head:
  num_convs: 4
  num_filters: 256
  use_separable_conv: false
retinanet_loss:
  box_loss_weight: 50
  focal_loss_alpha: 0.25
  focal_loss_gamma: 1.5
  huber_loss_delta: 0.1
retinanet_parser:
  aug_rand_hflip: true
  aug_scale_max: 1.0
  aug_scale_min: 1.0
  autoaugment_policy_name: v0
  match_threshold: 0.5
  max_num_instances: 100
  num_channels: 3
  output_size: [640, 640]
  skip_crowd_during_training: true
  unmatched_threshold: 0.5
  use_autoaugment: false
spinenet:
  model_id: '49'
strategy_config:
  all_reduce_alg: null
  distribution_strategy: one_device
  num_gpus: 1
  num_packs: 1
  task_index: -1
  tpu: Quadro
  worker_hosts: null
strategy_type: one_device
train:
  batch_size: 64
  checkpoint:
    path: resnet50-2018-02-07
    prefix: resnet50/
  frozen_variable_prefix: ''
  gradient_clip_norm: 0.0
  input_partition_dims: null
  input_sharding: false
  iterations_per_loop: 100
  l2_weight_decay: 0.0001
  learning_rate:
    init_learning_rate: 0.08
    learning_rate_levels: [0.008, 0.0008]
    learning_rate_steps: [15000, 20000]
    type: step
    warmup_learning_rate: 0.0067
    warmup_steps: 500
  num_cores_per_replica: null
  optimizer:
    momentum: 0.9
    nesterov: true
    type: momentum
  regularization_variable_regex: .*(kernel|weight):0$
  total_steps: 22500
  train_dataset_type: tfrecord
  train_file_pattern: annotations\train.record
  transpose_input: false
type: retinanet
use_tpu: false

How do I change the parameters other than default ones provided in --params_override ?

Appendix - Full Error Message:

Traceback (most recent call last):
  File "models/official/vision/detection/main.py", line 265, in <module>
    app.run(main)
  File "C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tf-official\lib\site-packages\absl\app.py", line 303, in run
    _run_main(main, args)
  File "C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tf-official\lib\site-packages\absl\app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "models/official/vision/detection/main.py", line 260, in main
    run()
  File "models/official/vision/detection/main.py", line 185, in run
    params = params_dict.override_params_dict(
  File "C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tf-official\lib\site-packages\official\modeling\hyperparams\params_dict.py", line 443, in override_params_dict
    params.override(params_dict, is_strict)
  File "C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tf-official\lib\site-packages\official\modeling\hyperparams\params_dict.py", line 166, in override
    self._override(override_params, is_strict)  # pylint: disable=protected-access
  File "C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tf-official\lib\site-packages\official\modeling\hyperparams\params_dict.py", line 183, in _override
    self.__dict__[k]._override(v, is_strict)  # pylint: disable=protected-access
  File "C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tf-official\lib\site-packages\official\modeling\hyperparams\params_dict.py", line 176, in _override
    raise KeyError('The key `{}` does not exist. '
KeyError: 'The key `total_steps:100` does not exist. To extend the existing keys, use `override` with `is_strict` = False.'

Answer 1

I saw the additional instructions a little bit further down the README document.

1. You can create a YAML config file along with the command, eg my_retinanet.yaml. This file specifies the parameters to be overridden, which should at least include the following fields.

python3 ~/models/official/vision/detection/main.py \
  --strategy_type=tpu \
  --tpu="${TPU_NAME?}" \
  --model_dir="${MODEL_DIR?}" \
  --mode=train \
  --config_file="my_retinanet.yaml"

With the config file as:

# my_retinanet.yaml
type: 'retinanet'
train:
  train_file_pattern: <path to the TFRecord training data>
eval:
  eval_file_pattern: <path to the TFRecord validation data>
  val_json_file: <path to the validation annotation JSON file>

Or 2) You can use inline configuration (YAML or JSON format):

python3 ~/models/official/vision/detection/main.py \
  --model_dir=<model folder> \
  --strategy_type=one_device \
  --num_gpus=1 \
  --mode=train \
  --params_override="eval:
 eval_file_pattern: <Eval TFRecord file pattern>
 batch_size: 8
 val_json_file: <COCO format groundtruth JSON file>
predict:
 predict_batch_size: 8
architecture:
 use_bfloat16: False
train:
 total_steps: 1
 batch_size: 8
 train_file_pattern: <Eval TFRecord file pattern>
use_tpu: False
"

For some reason, the JSON inline format didn't work for me, but the my_retinanet.yaml file did for me.