tensorflow.python.framework.errors_impl.InvalidArgumentError：断言失败：

Question

2022-08-17 09:50:13.773944: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-17 09:50:13.775825: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-17 09:50:13.779061: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-08-17 09:50:13.782886: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-17 09:50:13.784665: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-17 09:50:13.786370: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-17 09:50:15.199944: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-17 09:50:15.201818: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-17 09:50:15.203250: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-17 09:50:15.204640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38268 MB memory:  -> device: 0, name: NVIDIA A100-PCIE-40GB, pci bus id: 0000:00:05.0, compute capability: 8.0
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
I0817 09:50:17.109207 140027332416448 mirrored_strategy.py:374] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
INFO:tensorflow:Maybe overwriting train_steps: 5000
I0817 09:50:17.115118 140027332416448 config_util.py:552] Maybe overwriting train_steps: 5000
INFO:tensorflow:Maybe overwriting use_bfloat16: False
I0817 09:50:17.115335 140027332416448 config_util.py:552] Maybe overwriting use_bfloat16: False
WARNING:tensorflow:From /home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/object_detection/model_lib_v2.py:563: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version.
Instructions for updating:
rename to distribute_datasets_from_function
W0817 09:50:17.151397 140027332416448 deprecation.py:350] From /home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/object_detection/model_lib_v2.py:563: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version.
Instructions for updating:
rename to distribute_datasets_from_function
INFO:tensorflow:Reading unweighted datasets: ['Tensorflow/workspace/annotations/train.record']
I0817 09:50:17.155634 140027332416448 dataset_builder.py:162] Reading unweighted datasets: ['Tensorflow/workspace/annotations/train.record']
INFO:tensorflow:Reading record datasets for input file: ['Tensorflow/workspace/annotations/train.record']
I0817 09:50:17.155876 140027332416448 dataset_builder.py:79] Reading record datasets for input file: ['Tensorflow/workspace/annotations/train.record']
INFO:tensorflow:Number of filenames to read: 1
I0817 09:50:17.155959 140027332416448 dataset_builder.py:80] Number of filenames to read: 1
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
W0817 09:50:17.156022 140027332416448 dataset_builder.py:86] num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From /home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/object_detection/builders/dataset_builder.py:100: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.deterministic`.
W0817 09:50:17.158816 140027332416448 deprecation.py:350] From /home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/object_detection/builders/dataset_builder.py:100: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.deterministic`.
WARNING:tensorflow:From /home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/object_detection/builders/dataset_builder.py:235: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
W0817 09:50:17.187078 140027332416448 deprecation.py:350] From /home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/object_detection/builders/dataset_builder.py:235: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
WARNING:tensorflow:From /home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/tensorflow/python/util/dispatch.py:1082: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.
W0817 09:50:24.743995 140027332416448 deprecation.py:350] From /home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/tensorflow/python/util/dispatch.py:1082: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.
WARNING:tensorflow:From /home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/tensorflow/python/util/dispatch.py:1082: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version.
Instructions for updating:
`seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead.
W0817 09:50:28.231675 140027332416448 deprecation.py:350] From /home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/tensorflow/python/util/dispatch.py:1082: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version.
Instructions for updating:
`seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead.
WARNING:tensorflow:From /home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/tensorflow/python/util/dispatch.py:1082: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
W0817 09:50:29.841259 140027332416448 deprecation.py:350] From /home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/tensorflow/python/util/dispatch.py:1082: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
Traceback (most recent call last):
  File "/home/thecon/Documents/AiMedia/TF-Detection/Tensorflow/models/research/object_detection/model_main_tf2.py", line 114, in <module>
    tf.compat.v1.app.run()
  File "/home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/tensorflow/python/platform/app.py", line 36, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "/home/thecon/Documents/AiMedia/TF-Detection/Tensorflow/models/research/object_detection/model_main_tf2.py", line 105, in main
    model_lib_v2.train_loop(
  File "/home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/object_detection/model_lib_v2.py", line 605, in train_loop
    load_fine_tune_checkpoint(
  File "/home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/object_detection/model_lib_v2.py", line 401, in load_fine_tune_checkpoint
    _ensure_model_is_built(model, input_dataset, unpad_groundtruth_tensors)
  File "/home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/object_detection/model_lib_v2.py", line 161, in _ensure_model_is_built
    features, labels = iter(input_dataset).next()
  File "/home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/tensorflow/python/distribute/input_lib.py", line 569, in next
    return self.__next__()
  File "/home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/tensorflow/python/distribute/input_lib.py", line 573, in __next__
    return self.get_next()
  File "/home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/tensorflow/python/distribute/input_lib.py", line 630, in get_next
    return self._get_next_no_partial_batch_handling(name)
  File "/home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/tensorflow/python/distribute/input_lib.py", line 662, in _get_next_no_partial_batch_handling
    replicas.extend(self._iterators[i].get_next_as_list(new_name))
  File "/home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/tensorflow/python/distribute/input_lib.py", line 1632, in get_next_as_list
    return self._format_data_list_with_options(self._iterator.get_next())
  File "/home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/tensorflow/python/data/ops/multi_device_iterator_ops.py", line 531, in get_next
    result.append(self._device_iterators[i].get_next())
  File "/home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 819, in get_next
    return self._next_internal()
  File "/home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 749, in _next_internal
    ret = gen_dataset_ops.iterator_get_next(
  File "/home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 3017, in iterator_get_next
    _ops.raise_from_not_ok_status(e, name)
  File "/home/thecon/miniconda3/envs/tf/lib/python3.10/site-packages/tensorflow/python/framework/ops.py", line 7164, in raise_from_not_ok_status
    raise core._status_to_exception(e) from None  # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [[0.866666675]] [[0.673333347]]
         [[{{function_node Assert_AssertGuard_false_856}}{{node Assert/AssertGuard/Assert}}]]
         [[MultiDeviceIteratorGetNextFromShard]]
         [[RemoteCall]] [Op:IteratorGetNext]

My train process stuck here. This error came from nowhere, I just regenerate tfrecords and it appeared. Untill this error appears, the process fronzez at "Instructions for updating: Use tf.cast instead", with process still running but fronzen. I regenerated the tfrecords because has some problems on call function paths, and now I get this errror.

Answer 1

The reason for the error was the bad assertion to csv collons for xmin, xmax, ymin, ymax, the order didn't corespond to the xml.