AWS Glue : How to make sure glue crawler always picks up the latest file from S3

Question

I have an ETL pipeline which outputs a.csv files into S3 every 15 minutes. How can I configure a glue crawler such that it picks up only the latest file instead of using all the files.

Answer 1

Using incremental crawls:

For an Amazon Simple Storage Service (Amazon S3) data source, incremental crawls only crawl folders that were added since the last crawler run. Without this option, the crawler crawls the entire dataset. ... To perform an incremental crawl, you can set the Crawl new folders only option in the AWS Glue console or set the RecrawlPolicy property in the CreateCrawler request in the API.