Aws Glue Crawler 在第一次爬网后没有更新表

Question

I am adding a new file in parquet format which is created by a Glue Databrew in my S3 folder. The new file has the same schema as the previous file. But when I am running the Crawler for the 2nd time it is neither updating the table nor creating a new one in the data catalog. Also when I am crawling both the files together, both of them are getting added.

Log File is giving the following information:
INFO: Created partitions with values [[New file name]] for table
BENCHMARK: Finished writing to Catalog

I have tried with and without "Create a single schema for each S3 path". But the crawler is not updating the table with the new file. Sooner I will add new files on a daily basis to do my analysis. Any solution?

Answer 1

The best way to approach this issue in my opinion is to use AWS DataBrew output to Data Catalog directly. Data Catalog can be updated either by the crawler or by DataBrew directly but the recommended practice is that you employ any one of those mechanisms not both.

Can you try running the job with output as your data catalog and let Databrew manage your catalog? It should update your catalog table with right data/files.