在雅典娜查詢boto3之后，無法將文件從一個s3位置復制到另一位置

Question

我正在使用boto3查詢雅典娜的結果。 他們工作正常。 我再次使用boto3將文件從一個s3存儲桶復制到另一個s3存儲桶，但是它說無法找到該文件。 我找不到解決方案。 請幫忙！

當我轉到s3控制台時，我可以看到該文件，但是boto3找不到它。

import boto3

athena = boto3.client('athena')
s3 = boto3.resource('s3')
BUCKET_NAME = 'bucket1'
bucket = s3.Bucket(BUCKET_NAME)


query = 'SELECT * FROM "db"."table" limit 2'
response = athena.start_query_execution(QueryString=query, QueryExecutionContext={
            'Database': 'db'
            }, ResultConfiguration={
            'OutputLocation': 's3://bucket1/',
            })

key = response['QueryExecutionId'] + '.csv'

copy_source = {
    'Bucket': 'bucket1',
    'Key': key
}
s3.meta.client.copy(copy_source, 'bucket2', 'main.csv')

錯誤是：-

Traceback (most recent call last):
  File "/Users/tanmaysinghal/Vizualization/Python Scripts/test.py", line 23, in <module>
    s3.meta.client.copy(copy_source, 'bucket2', 'main.csv')
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/boto3/s3/inject.py", line 379, in copy
    return future.result()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/s3transfer/tasks.py", line 255, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/s3transfer/copies.py", line 110, in _submit
    **head_object_request)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found

Answer 1

您的代碼存在根本問題，即您正在嘗試在查詢完成之前復制Athena S3 output文件。 當您在控制台中看到該文件時，可能要花幾秒鍾才能看到文件，此時該文件已准備就緒。 您必須等待Athena完成查詢並將output寫入S3 。

雅典娜的運作方式如下：

1. 提交查詢。
2. 檢查查詢status 。
3. 如果狀態running ，等待，轉到步驟2 ELSEIF SUCCEEDED ，轉到步驟4.否則失敗了，那么做的糾正措施。
4. 讀取輸出的S3 file 。

這里是工作代碼。

import boto3
import time
athena = boto3.client('athena')
# s3 = boto3.resource('s3')
# BUCKET_NAME = 'bucket1'
# bucket = s3.Bucket(BUCKET_NAME)

query = 'SELECT *  FROM your-database.your-table  limit 10'
response = athena.start_query_execution(QueryString=query, QueryExecutionContext={
            'Database': 'your-database'
            }, ResultConfiguration={
            'OutputLocation': 's3://your-s3-output-bucket',
            })

execution_id = response['QueryExecutionId']

key = execution_id + ".csv"
state = 'RUNNING';
##waiting for query to complete and then read the result.
while (state in ['RUNNING']):
    response = athena.get_query_execution(QueryExecutionId=execution_id)
    if 'QueryExecution' in response and \
                'Status' in response['QueryExecution'] and \
                'State' in response['QueryExecution']['Status']:
            state = response['QueryExecution']['Status']['State']
            if state == 'FAILED':
                print("FAILED")
            elif state == 'SUCCEEDED':
                s3_path = response['QueryExecution']['ResultConfiguration']['OutputLocation']
                print("S3-Path:" + s3_path)
    time.sleep(1)

# If state is succeeded, meaning query has completed successfully, now, you could read the output file or try copy it to somewhere else.
if state == 'SUCCEEDED':
    copy_source = {
        'Bucket': 'your-s3-output-bucket',
        'Key': key
    }
    s3.meta.client.copy(copy_source, 'bucket2', 'main.csv')

希望對您有幫助！