tensorboard caffe2 can't find blob

Question

I'm trying to record some values with tensorboard writer with the following lines. If I leave the commented section commented, then the code runs fine. But when I uncomment it, I get an error. Any advice would be appreciated.

if (tb_writer != None):
        for key, value in qps.items():
            tb_writer.add_scalar("QPv2_" + mode + "_Threshold_" + str(key), value, global_step)
        # for annotation_idx in range(1, args.num_classes-1):
        #     annotation_class = annotation_classes[annotation_idx]
        #     for threshold_idx in range(len(annotation_thresholds)):
        #         threshold = annotation_thresholds[threshold_idx]
        #         tb_writer.add_scalar(annotation_class + "_Threshold_" + str(threshold) + "_Precision", str(precision[threshold_idx][annotation_idx].item()), global_step)
        #         tb_writer.add_scalar(annotation_class + "_Threshold_" + str(threshold) + "_Recall", str(recall[threshold_idx][annotation_idx].item()), global_step)

Traceback (most recent call last):
  File "train_transformer_v5.py", line 782, in <module>
    main()
  File "train_transformer_v5.py", line 763, in main
    global_step, tr_loss, metrics_result = trainer.train(args)
  File "train_transformer_v5.py", line 451, in train
    tb_writer=self.tb_writer, global_step=self.global_step)
  File "/home/xianx/model/metrics_v5.py", line 606, in eval_metrics
    tb_writer.add_scalar(annotation_class + "_Threshold_" + str(threshold) + "_Precision", str(precision[threshold_idx][annotation_idx].item()), global_step)
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/tensorboard/writer.py", line 348, in add_scalar
    scalar_value = workspace.FetchBlob(scalar_value)
  File "/opt/conda/lib/python3.6/site-packages/caffe2/python/workspace.py", line 379, in FetchBlob
    result = C.fetch_blob(StringifyBlobName(name))
RuntimeError: [enforce fail at pybind_state.cc:221] ws->HasBlob(name). Can't find blob: 0.7158585786819458

Answer 1

My issue was that I was using .add_scalar(...) while converting my value to a string. Removing that conversion fixed the problem.