由于时间问题，Athena 查询返回空结果

Question

I'm trying to create and query the Athena table based on data located in S3, and it seems that there are some timing issues.

How can I know when all the partitions have been loaded to the table?

The following code returns an empty result -

athena_client.start_query_execution(QueryString=app_query_create_table,
                                    ResultConfiguration={'OutputLocation': output_location})

athena_client.start_query_execution(QueryString="MSCK REPAIR TABLE `{athena_db}`.`{athena_db_partition}`"
                .format(athena_db=athena_db, athena_db_partition=athena_db_partition),
            ResultConfiguration={'OutputLocation': output_location})

result = query.format(athena_db_partition=athena_db_partition, delta=delta, dt=dt)

But when I add some delay, it works greate -

athena_client.start_query_execution(QueryString=app_query_create_table,
                                    ResultConfiguration={'OutputLocation': output_location})

athena_client.start_query_execution(QueryString="MSCK REPAIR TABLE `{athena_db}`.`{athena_db_partition}`"
                .format(athena_db=athena_db, athena_db_partition=athena_db_partition),
            ResultConfiguration={'OutputLocation': output_location})

time.sleep(3)

result = query.format(athena_db_partition=athena_db_partition, delta=delta, dt=dt)

The following is the query for creating the table -

query_create_table = '''
            CREATE EXTERNAL TABLE `{athena_db}`.`{athena_db_partition}` (
                `time` string,
                `user_advertiser_id` string,
                `predictions` float
            ) PARTITIONED BY (
                dt string
            )
            ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
            WITH SERDEPROPERTIES (
                'serialization.format' = ',',
                'field.delim' = ','
            ) LOCATION 's3://{bucket}/path/'
            '''

app_query_create_table = query_create_table.format(bucket=bucket,
                                                   athena_db=athena_db,
                                                     athena_db_partition=athena_db_partition)

I would love to get some help.

Answer 1

The start_query_execution call only starts the query, it does not wait for it to complete. You must run get_query_execution periodically until the status of the execution is successful (or failed).

---

Not related to your problem per se, but if you create a table with CREATE TABLE … AS there is no need to add partitions with MSCK REPAIR TABLE … afterwards, there will be no new partitions after the table has just been created that way – because it will be created with all the partitions produced by the query.

Also, in general, avoid using MSCK REPAIR TABLE , it is slow and inefficient. There are many better ways to add partitions to a table, see https://athena.guide/articles/five-ways-to-add-partitions/