[英]Airflow mysql to google cloud storage, byte-like object required error
I am working on a DAG that queries a MySQL database, extracts and loads the data to Google Cloud Storage. 我正在研究查询MySQL数据库,提取数据并将其加载到Google Cloud Storage的DAG。
The table that I am trying to export includes text, int, float, varchar(20) and varchar(32) data. 我尝试导出的表包括文本,整型,浮点型,varchar(20)和varchar(32)数据。
I am using airflow v1.8.0. 我正在使用气流v1.8.0。
default_args = {
'owner' : 'tia',
'start_date' : datetime(2018, 1, 4),
'depends_on_past' : False,
'retries' : 1,
'retry_delay':timedelta(minutes=5),
}
dag = DAG('mysql_to_gcs', default_args=default_args)
export_waybills = MySqlToGoogleCloudStorageOperator(
task_id='extract_waybills',
mysql_conn_id = 'podiotestmySQL',
sql = 'SELECT * FROM podiodb.logistics_waybills',
bucket='podio-reader-storage',
filename= 'podio-data/waybills{}.json',
schema_filename='podio-data/schema/waybills.json',
dag=dag)
I cam across the following error, which seems to be similar to this Airflow mysql to gcp Dag error 我凸轮遇到以下错误,这似乎类似于此Airflow mysql to gcp Dag错误
[2018-01-04 11:12:23,372] {models.py:1342} INFO - Executing on 2018-01-04 00:00:00
[2018-01-04 11:12:23,372] {models.py:1342}信息-在2018-01-04 00:00:00执行
[2018-01-04 11:12:23,400] {base_hook.py:67} INFO - Using connection to: 35.189.207.140
[2018-01-04 11:12:23,400] {base_hook.py:67}信息-使用连接到:35.189.207.140
[2018-01-04 11:12:24,903] {models.py:1417} ERROR - a bytes-like object is required, not 'str'
[2018-01-04 11:12:24,903] {models.py:1417}错误-需要类似字节的对象,而不是'str'
Traceback (most recent call last):
追溯(最近一次通话):
File "/home/hyperli/.local/lib/python3.5/site-packages/airflow/models.py", line 1374, in run
运行中的文件“ /home/hyperli/.local/lib/python3.5/site-packages/airflow/models.py”,行1374
result = task_copy.execute(context=context)
结果= task_copy.execute(context = context)
File "/home/hyperli/.local/lib/python3.5/site-packages/airflow/contrib/operators/mysql_to_gcs.py", line 91, in execute
执行中的文件“ /home/hyperli/.local/lib/python3.5/site-packages/airflow/contrib/operators/mysql_to_gcs.py”,第91行
files_to_upload = self._write_local_data_files(cursor)
files_to_upload = self._write_local_data_files(光标)
File "/home/hyperli/.local/lib/python3.5/site-packages/airflow/contrib/operators/mysql_to_gcs.py", line 136, in _write_local_data_files
_write_local_data_files中的文件“ /home/hyperli/.local/lib/python3.5/site-packages/airflow/contrib/operators/mysql_to_gcs.py”,第136行
json.dump(row_dict, tmp_file_handle)
json.dump(row_dict,tmp_file_handle)
File "/usr/lib/python3.5/json/ init .py", line 179, in dump fp.write(chunk)
转储文件fp.write(chunk)中的文件“ /usr/lib/python3.5/json/ init .py”,行179
File "/usr/lib/python3.5/tempfile.py", line 622, in func_wrapper return func(*args, **kwargs)
在func_wrapper中的文件“ /usr/lib/python3.5/tempfile.py”,第622行,返回func(* args,** kwargs)
TypeError: a bytes-like object is required, not 'str'
TypeError:需要一个类似字节的对象,而不是'str'
[2018-01-04 11:12:24,907] {models.py:1433} INFO - Marking task as UP_FOR_RETRY
[2018-01-04 11:12:24,907] {models.py:1433}信息-将任务标记为UP_FOR_RETRY
[2018-01-04 11:12:25,037] {models.py:1462} ERROR - a bytes-like object is required, not 'str'
[2018-01-04 11:12:25,037] {models.py:1462}错误-需要类似字节的对象,而不是'str'
Does anyone know why the exception is thrown? 有谁知道为什么会引发异常?
Are you using python 3? 您在使用python 3吗? Because it seams mysql_to_gcs of the latest version (1.90) is not python compatible it seams.
因为它接缝了最新版本(1.90)的mysql_to_gcs与python不兼容,所以它接缝了。
It seams that this operator was changed here to make mysql_to_gcs py3 compatible. 似乎在此更改了该运算符以使mysql_to_gcs py3兼容。 But the latest version(1.90) doesn't include this change
但是最新版本(1.90)不包含此更改
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.