如何在 PyAthena 中處理錯誤並重試？

Question

我有一個每天從本地 Ubuntu 機器運行的 Athena 查詢。 大多數時候它運行良好。

def get_athena_data(**kwargs):
   athena_conn = connect(aws_access_key_id = access_key, aws_secret_access_key = s_key, s3_staging_dir = path, region_name = region)
   print(f"{datetime.today().strftime('%Y-%m-%d %H:%M.%S')} Athena connection established; starting to query data using pd-sql integration")
   load_data = pd.read_sql(sql,athena_conn)
   return load_data

然而，前幾天我得到了（這很長，所以我用了幾次 SNIP）：

Traceback (most recent call last):
  File "/home/ken/anaconda3/lib/python3.7/site-packages/pandas/io/sql.py", line 1586, in execute
    cur.execute(*args, **kwargs)
  File "/home/ken/anaconda3/lib/python3.7/site-packages/pyathena/util.py", line 306, in _wrapper
    return wrapped(*args, **kwargs)
  File "/home/ken/anaconda3/lib/python3.7/site-packages/pyathena/cursor.py", line 79, in execute
    raise OperationalError(query_execution.state_change_reason)
pyathena.error.OperationalError: GENERIC_INTERNAL_ERROR: Unable to create class ... SNIP ...
]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ken/anaconda3/lib/python3.7/site-packages/pandas/io/sql.py", line 1590, in execute
    self.con.rollback()
  File "/home/ken/anaconda3/lib/python3.7/site-packages/pyathena/connection.py", line 184, in rollback
    raise NotSupportedError
pyathena.error.NotSupportedError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ken/Documents/projects/site_alerts/code/Site_alerts_v18.py", line 174, in <module>
    sql_results = get_athena_data(SNIP)
  File "/home/ken/Documents/projects/site_alerts/code/site_alert_functions_v18.py", line 351, in get_athena_data
    load_data = pd.read_sql(sql,athena_conn)
  File "/home/ken/anaconda3/lib/python3.7/site-packages/pandas/io/sql.py", line 412, in read_sql
    chunksize=chunksize,
  File "/home/ken/anaconda3/lib/python3.7/site-packages/pandas/io/sql.py", line 1633, in read_query
    cursor = self.execute(*args)
  File "/home/ken/anaconda3/lib/python3.7/site-packages/pandas/io/sql.py", line 1595, in execute
    raise ex from inner_exc
pandas.io.sql.DatabaseError: Execution failed on sql: ... SNIP ... 


GENERIC_INTERNAL_ERROR: Unable to create class ... SNIP ...
]
unable to rollback

很好，所以我需要處理錯誤，如果它確實失敗了，我想重試。 所以我嘗試了：

def retry(func, max_tries=5):
    for i in range(max_tries):
        try:
           func()
           print('completed successfully')
           break
        except Exception:
           print('error')
           continue

retry(get_athena_data(ARGS))

但是，這是行不通的。 當 Athena 失敗時，它仍然會停止執行（我輸入了一個有缺陷的 sql 查詢來模擬）。

如何處理異常並執行重試？

我在 Pyathena 問題中發現了這一點，但這對我來說毫無意義，也沒有使用說明。

Answer 1

您正在調用 function get_athena_data並將其返回傳遞給 function retry ，而不是 function。

試試這種方式： retry(get_athena_data) 。

（更新）現在傳遞一些參數：

def retry(func, max_tries=5, *args, **kwargs):
    for i in range(max_tries):
        try:
           func(*args, **kwargs)
           print('completed successfully')
           break
        except Exception:
           print('error')
           continue

retry(get_athena_data, arg1, arg2, kwarg1="foo", kwarg2="bar")