ml_metadata.errors.AlreadyExistsError：给定节点已经存在

Question

我在使用 TFX、MLMD 和 Apache-Airflow 作为编排器时遇到了问题。 由 TFX 提供的 Local-dag-runner 工作正常，导致每个管道组件运行时产生不同的工件。 当 airflow 用作编排器时，就会出现问题。 如果您只是重新运行管道而不对代码进行任何更改，则管道运行时不会出错。 但是，当您更改某些参数时（例如，在 Tuner 的配置文件中），管道无法完成运行。 这是错误消息：

ml_metadata.errors.AlreadyExistsError：给定节点已经存在：type_id：20

这里几乎是整个日志：

> AIRFLOW_CTX_DAG_OWNER=airflow
> AIRFLOW_CTX_DAG_ID=consumer_complaint_pipeline_airflow
> AIRFLOW_CTX_TASK_ID=Transform
> AIRFLOW_CTX_EXECUTION_DATE=2022-07-12T10:25:40.659461+00:00 AIRFLOW_CTX_TRY_NUMBER=1 AIRFLOW_CTX_DAG_RUN_ID=manual__2022-07-12T10:25:40.659461+00:00 [2022-07-12 14:57:12,605] {base_component_launcher.py:189} INFO - Running driver for Transform [2022-07-12 14:57:12,612] {metadata_store.py:105} INFO - MetadataStore with DB connection initialized [2022-07-12 14:57:12,800] {taskinstance.py:1889} ERROR - Task failed with exception

> Traceback (most recent call last):   File "/home/mlops/GIT_REPO/davari/lib/python3.9/site-packages/airflow/operators/python.py", line 171, in execute
    return_value = self.execute_callable()   File "/home/mlops/GIT_REPO/davari/lib/python3.9/site-packages/airflow/operators/python.py", line 189, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)   File "/home/mlops/GIT_REPO/davari/lib/python3.9/site-packages/tfx/orchestration/airflow/airflow_component.py", line 76, in _airflow_component_launcher
    launcher.launch()   File "/home/mlops/GIT_REPO/davari/lib/python3.9/site-packages/tfx/orchestration/launcher/base_component_launcher.py", line 191, in launch
    execution_decision = self._run_driver(self._input_dict, self._output_dict,   File "/home/mlops/GIT_REPO/davari/lib/python3.9/site-packages/tfx/orchestration/launcher/base_component_launcher.py", line 155, in _run_driver
    execution_decision = driver.pre_execution(   File "/home/mlops/GIT_REPO/davari/lib/python3.9/site-packages/tfx/dsl/components/base/base_driver.py", line 320, in pre_execution
    self._metadata_handler.update_execution(   File "/home/mlops/GIT_REPO/davari/lib/python3.9/site-packages/tfx/orchestration/metadata.py", line 651, in update_execution
    _, a_ids, _ = self.store.put_execution(execution, artifacts_and_events,   File "/home/mlops/GIT_REPO/davari/lib/python3.9/site-packages/ml_metadata/metadata_store/metadata_store.py", line 591, in put_execution
    self._call('PutExecution', request, response)   File "/home/mlops/GIT_REPO/davari/lib/python3.9/site-packages/ml_metadata/metadata_store/metadata_store.py", line 188, in _call
    return self._call_method(method_name, request, response)   File "/home/mlops/GIT_REPO/davari/lib/python3.9/site-packages/ml_metadata/metadata_store/metadata_store.py", line 209, in _call_method
    self._pywrap_cc_call(cc_method, request, response)   File "/home/mlops/GIT_REPO/davari/lib/python3.9/site-packages/ml_metadata/metadata_store/metadata_store.py", line 240, in _pywrap_cc_call
    raise _make_exception(error_message.decode('utf-8'), status_code)

> ml_metadata.errors.AlreadyExistsError: Given node already exists: type_id: 20 uri: "/home/mlops/airflow/tfx/consumer_complaint_pipeline_airflow/Transform/transform_graph/24"

> custom_properties {   key: "name"   value {
    string_value: "transform_graph"   } } custom_properties {   key: "producer_component"   value {
    string_value: "Transform"   } } name: "transform_graph"

> INTERNAL: Cannot create node for type_id: 20 uri:

---

编辑

Answer 1

已提交PR以解决工件命名问题。 合并 PR 后，您可以尝试让我们知道问题是否仍然存在。 谢谢