aws胶水触发作业

Question

I have modified a Glue generated script that I use for transformation and manipulation with the data. I want to run the same job by trigger on every new table that appears in the catalog but without manually changing the table name in the job script. In short, how can I run the same transformation the script provides on every new table that appears in the data catalog without manually changing the table name every time ?

Thanks

Answer 1

You can use Catalog Client to dynamically get the list of tables in the database. I don't know how to get the catalog client in pyspark, but in scala it looks like this

val catalog = glueContext.getCatalogClient

for (table <- catalog.listTables("myDatabaseName", "").getTableList.asScala) {
    // do your transformation
}