dtype=datetime64[ns, UTC-03:00] 和 datetime 之间的比较无效

Question

这个问题发生在一个etl creation中，这是一个日期转换的问题，使用panda。 有人能帮我吗？

df_delivered = df.loc[(df['state'] == 2) | (df['state'] == 4) | (df['state'] == 7) | (df['direction'] == 1) | (df['delivered_at'] > last_update)]
df_delivered['Delivered'] = 1
df1 = df.merge(df_delivered, left_index=True, right_index=True, how='outer', suffixes=('', '_y'))
df1.drop(df1.filter(regex='_y$').columns, axis=1, inplace=True)
df = df1

并收到此错误：

上述异常是以下异常的直接原因：

Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.9/site-packages/pandas/core/arrays/datetimelike.py", line 1008, in \_cmp_method
other = self.\_validate_comparison_value(other)
File "/home/airflow/.local/lib/python3.9/site-packages/pandas/core/arrays/datetimelike.py", line 543, in \_validate_comparison_value
raise InvalidComparison(other) from err
pandas.core.arrays.datetimelike.InvalidComparison: 2019-01-01 00:00:00

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/task/task_runner/standard_task_runner.py", line 85, in \_start_by_fork
args.func(args, dag=self.dag)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(\*args, \*\*kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/cli.py", line 92, in wrapper
return f(\*args, \*\*kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/cli/commands/task_command.py", line 292, in task_run
\_run_task_by_selected_method(args, dag, ti)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/cli/commands/task_command.py", line 107, in \_run_task_by_selected_method
\_run_raw_task(args, ti)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/cli/commands/task_command.py", line 180, in \_run_raw_task
ti.\_run_raw_task(
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/session.py", line 70, in wrapper
return func(\*args, session=session, \*\*kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1332, in \_run_raw_task
self.\_execute_task_with_callbacks(context)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1458, in \_execute_task_with_callbacks
result = self.\_execute_task(context, self.task)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1509, in \_execute_task
result = execute_callable(context=context)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/operators/python.py", line 151, in execute
return_value = self.execute_callable()
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/operators/python.py", line 162, in execute_callable
return self.python_callable(\*self.op_args, \*\*self.op_kwargs)
File "/opt/airflow/etl/dw_reports/etl_communication_customer_engagement.py", line 62, in etl_communication_customer_engagement
df_delivered = df.loc\[(df\['state'\] == 2) | (df\['state'\] == 4) | (df\['state'\] == 7) | (df\['direction'\] == 1) | (df\['delivered_at'\] \> last_update)\]
File "/home/airflow/.local/lib/python3.9/site-packages/pandas/core/ops/common.py", line 69, in new_method
return method(self, other)
File "/home/airflow/.local/lib/python3.9/site-packages/pandas/core/arraylike.py", line 48, in __gt__
return self.\_cmp_method(other, operator.gt)
File "/home/airflow/.local/lib/python3.9/site-packages/pandas/core/series.py", line 5502, in \_cmp_method
res_values = ops.comparison_op(lvalues, rvalues, op)
File "/home/airflow/.local/lib/python3.9/site-packages/pandas/core/ops/array_ops.py", line 270, in comparison_op
res_values = op(lvalues, rvalues)
File "/home/airflow/.local/lib/python3.9/site-packages/pandas/core/ops/common.py", line 69, in new_method
return method(self, other)
File "/home/airflow/.local/lib/python3.9/site-packages/pandas/core/arraylike.py", line 48, in __gt__
return self.\_cmp_method(other, operator.gt)
File "/home/airflow/.local/lib/python3.9/site-packages/pandas/core/arrays/datetimelike.py", line 1010, in \_cmp_method
return invalid_comparison(self, other, op)
File "/home/airflow/.local/lib/python3.9/site-packages/pandas/core/ops/invalid.py", line 34, in invalid_comparison
raise TypeError(f"Invalid comparison between dtype={left.dtype} and {typ}")
TypeError: Invalid comparison between dtype=datetime64\[ns, UTC-03:00\] and datetime
\[2023-01-09, 13:54:11 UTC\] {local_task_job.py:154} INFO - Task exited with return code 1
\[2023-01-09, 13:54:11 UTC\] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule check\`

Answer 1

似乎这种比较可能有问题：

(df['delivered_at'] > last_update)

如果此处的 last_update 是传递值，则它需要与 df['delivered_at'] 列中的数据具有相同/可比较的类型。

顺便提一句。 这可以通过以下方式实现：

df.loc[:, ('delivered_at')] = pandas.to_datetime(df.loc[:, ('delivered_at')])