AWS glue Workers vs Concurrency

Question

I am looking for suggestions on how to set glue job parameters. Currently I get "Concurrent runs exceeded"as exception.

What can i do to scale and run the job properly

Answer 1

Concurrent runs exceeded means that you tried to run the job while an instance of the job is already running and you set the MaxConcurrentRuns property of the job. You can scale a job not only by running it concurrently, which I also would advise to, since this most of the Databases / Filesystems you are reading and writing to and from are probably not ACID compliant, you can also increase the NumberOfWorkers to deploy additional executor nodes.