Lambda function to query AWS Athena gives timeout

Question

I have written a Lambda function using athena-express that queries AWS Athena with S3 Parquet files as destination. Also using AWS Glue for ETL processes to S3. I am receiving: " Task timed out after 6.01 seconds ". Increasing the timeout gives me the same message but with higher amount of timeout seconds. The strange thing is that it works locally, but not when it's deployed to AWS. Code should work. I think something is missing in AWS configurations.

I have followed the instructions and using lambda settings: https://www.npmjs.com/package/athena-express . I have added AmazonAthenaFullAccess and AmazonS3FullAccess policies to the Execution role for the Lambda function.

Answer 1

When you say it works locally I assume that when you run it locally you have no timeout on the code. Athena is not a low latency database, and queries very rarely run in less than a few seconds, and often in minutes – it all depends on how much data they scan and other factors.

Your Lambda function must have a timeout that is longer than the longest run time of the queries you run with it. If those queries run for more than a minute, the timeout must be more than a minute, if they run for more than ten minutes, your timeout must be more than ten minutes, and so on.

In general, long running queries is not a great fit for Lambda and API Gateway that are made for more low latency situations. Since your queries seem to run for minutes I suggest that you look other ways of solving your problem. If you provide more context about what you are trying to do, we may be able to help you with that.