将每日快照数据从 AWS S3 移动到 AWS Athena

Question

Have a daily snapshot data that gets dumped into AWS S3 bucket. Each day's data file is build on top of previous day's file. I want to move this incremental data into AWS Athena without duplicating previous day's data.

I learned that AWS Glue can be a handy tool to move data into Athena on daily basis. but I am not sure how can i do it without duplicates.

Answer 1

When a table is defined in Amazon Athena (or AWS Glue), a location is provided that tells Athena where to look in Amazon S3 for the data.

This is conceptually different to a traditional database where data needs to be 'loaded'. No data needs to be 'loaded' or 'moved into' Amazon Athena -- it simply looks in the specified location and uses whatever data files it sees in that location (and any subdirectories).

If you are producing incremental files each day, then you can simply add additional files in that S3 location (making sure the filenames do not clash with existing files). Then, when a query is next run in Amazon Athena, those files will be included in the data that is scanned.

However, if you are producing a daily file with all data , then simply replace the previous file with the new file. Athena will use whatever file is in that location when running a query.