在S3分区上进行Hive查询太慢

Question

I have partitioned the data by date and here is how it is stored in s3.

s3://dataset/date=2018-04-01
s3://dataset/date=2018-04-02
s3://dataset/date=2018-04-03
s3://dataset/date=2018-04-04
...

Created hive external table on top of this. I am executing this query,

select count(*) from dataset where `date` ='2018-04-02' 

This partition has two parquet files like this,

part1 -xxxx- .snappy.parquet
part2 -xxxx- .snappy.parquet

each file size is 297MB. , So not a big file and not many files to scan.

And the query is returning 12201724 records. However it takes 3.5 mins to return this, since one partition itself is taking this time, running even the count query on whole dataset ( 7 years ) of data takes hours to return the results. Is there anyway, I can speed up this ?

Answer 1

Amazon Athena is, effectively, a managed Presto service. It can query data stored in Amazon S3 without having to run any clusters.

It is charged based upon the amount of data read from disk, so it runs very efficiently when using partitions and parquet files.

See: Analyzing Data in S3 using Amazon Athena | AWS Big Data Blog