AWS Lambda function fails while query Athena

Question

I am attempting to write a simple Lambda function to query a table in Athena. But after a few seconds I see "Status: FAILED" in the Cloudwatch logs. There is no descriptive error message on the cause of failure.

My test code is below:

import json
import time
import boto3

# athena constant
DATABASE = 'default'
TABLE = 'test'

# S3 constant
S3_OUTPUT = 's3://test-output/'

# number of retries
RETRY_COUNT = 1000

def lambda_handler(event, context):
    # created query
    query = "SELECT * FROM default.test limit 2"
    #  % (DATABASE, TABLE)
    # athena client
    client = boto3.client('athena')
    
    # Execution
    response = client.start_query_execution(
        QueryString=query,
        QueryExecutionContext={
            'Database': DATABASE
        },
        ResultConfiguration={
            'OutputLocation': S3_OUTPUT,
        }
    )
    
    # get query execution id
    query_execution_id = response['QueryExecutionId']
    print(query_execution_id)
    
    # get execution status
    for i in range(1, 1 + RETRY_COUNT):

        # get query execution
        query_status = client.get_query_execution(QueryExecutionId=query_execution_id)
        query_execution_status = query_status['QueryExecution']['Status']['State']

        if query_execution_status == 'SUCCEEDED':
            print("STATUS:" + query_execution_status)
            break

        if query_execution_status == 'FAILED':
            #raise Exception("STATUS:" + query_execution_status)
            print("STATUS:" + query_execution_status)

        else:
            print("STATUS:" + query_execution_status)
            time.sleep(i)
    else:
        # Did not encounter a break event. Need to kill the query
        client.stop_query_execution(QueryExecutionId=query_execution_id)
        raise Exception('TIME OVER')
        
    # get query results
    result = client.get_query_results(QueryExecutionId=query_execution_id)
    print(result)
    return

The logs show the following:

2020-08-31T10:52:12.443-04:00
    
START RequestId: e5434651-d36e-48f0-8f27-0290 Version: $LATEST
    
2020-08-31T10:52:13.481-04:00
    
88162f38-bfcb-40ae-b4a3-0b5a21846e28
    
2020-08-31T10:52:13.500-04:00
    
STATUS:QUEUED
    
2020-08-31T10:52:14.519-04:00
    
STATUS:RUNNING
    
2020-08-31T10:52:16.540-04:00
    
STATUS:RUNNING
    
2020-08-31T10:52:19.556-04:00
    
STATUS:RUNNING
    
2020-08-31T10:52:23.574-04:00
    
STATUS:RUNNING
    
2020-08-31T10:52:28.594-04:00
    
STATUS:FAILED
    
2020-08-31T10:52:28.640-04:00
    
....more status: FAILED
....
    
END RequestId: e5434651-d36e-48f0-8f27-0290
    
REPORT RequestId: e5434651-d36e-48f0-8f27-0290 Duration: 30030.22 ms Billed Duration: 30000 ms Memory Size: 128 MB Max Memory Used: 72 MB Init Duration: 307.49 ms

2020-08-31T14:52:42.473Z e5434651-d36e-48f0-8f27-0290 Task timed out after 30.03 seconds 

I think I have the right permissions for S3 bucket access given to the role (if not, I would have seen the error message). There are no files created in the bucket either. I am not sure what is going wrong here. What am I missing? Thanks

Answer 1

The last line in your log shows

2020-08-31T14:52:42.473Z e5434651-d36e-48f0-8f27-0290 Task timed out after 30.03 seconds 

To me this looks like the timeout of the Lambda Function is set to 30 seconds. Try increasing it to more than the time the Athena query needs (the maximum is 15 minutes).