AWS Glue中的ETL作业-是否可能覆盖数据？

Question

I am trying to write an AWS Glue ETL job that updates schema based on the most recent schema version.

I know this is not typically desirable behavior, but to minimize the number of output files, is it possible to do the transformations directly on the source data so that the transformed data is then loaded back to the same path?

Or is it possible to delete the data in the source path to then rewrite it to the same destination?

Answer 1

You do not need to ETL job to edit the schema unless if you want to automate the process. You can use edit schema feature of data catalog generated by AWS Glue crawler.

• Navigate to tables of AWS Glue
• Choose the table you want to change schema
• you can find edit schema button inside the table

After editing the schema, you can see the versions of the table auto maintained by the AWS Glue

Furthermore, you can also compare the versions of the table