AWS Data Wrangler - wr.athena.read_sql_query doesn't work

Question

I started using AWS Data Wrangler lib ( https://aws-data-wrangler.readthedocs.io/en/stable/what.html )

to execute queries on AWS Athena and use the results of them in my AWS Glue python shell job.

I see that exist wr.athena.read_sql_query to obtain what I need.

This is my code:

import sys
import os
import awswrangler as wr

os.environ['AWS_DEFAULT_REGION'] = 'eu-west-1'
databases = wr.catalog.databases()
print(databases)

query='select count(*) from staging_dim_channel'
print(query)
df_res = wr.athena.read_sql_query(sql=query, database="lsk2-target")
print(df_res)
print(f'DataScannedInBytes:            {df_res.query_metadata["Statistics"]["DataScannedInBytes"]}')
print(f'TotalExecutionTimeInMillis:    {df_res.query_metadata["Statistics"]["TotalExecutionTimeInMillis"]}')
print(f'QueryQueueTimeInMillis:        {df_res.query_metadata["Statistics"]["QueryQueueTimeInMillis"]}')
print(f'QueryPlanningTimeInMillis:     {df_res.query_metadata["Statistics"]["QueryPlanningTimeInMillis"]}')
print(f'ServiceProcessingTimeInMillis: {df_res.query_metadata["Statistics"]["ServiceProcessingTimeInMillis"]}')

I retrieve without problem the list of database (including the lsk2-target), but the read_sql_query go on error and I receive:

WaiterError: Waiter BucketExists failed: Max attempts exceeded

Please, can you help me to understand where I am wrong? Thanks!

Answer 1

Fixed a similar issue and the resolution is to ensure that the IAM role used has necessary Athena permission to create tables. As this API defaults to run in ctas_approach=True .

Ref.documentation

Also, once that is resolved ensure that the IAM role also has access to delete files create in S3

Answer 2

Do you have the right IAM permissions to read execute a query? I bet it is an IAM issue.

Also I guess you have setup your credentials:

[default]
aws_access_key_id = your_access_key_id
aws_secret_access_key = your_secret_access_key