How to stop an AWS Glue job gracefully?

Question

I am using Glue v3.0 (Python 3.7).

The goal is to stop the Glue PySpark job gracefully so that it can terminate properly and clean its custom resources before ending.

I tried to catch the stop/abort/kill signal, using the classical signal.signal() method with its handler.

However, it doesn't seem to work with the signals specified in the python signal's documentation (the handler never get called).

Any idea of how I can stop the job gracefully, using signal method or not?

Answer 1

You can call sys.exit(any_status_code) . Just make sure that sys.exit is called before job.commit(), otherwise the job will fail.

Just out of curiosity, why would you want to terminate your job by yourself? After you call job.commit() the job should end anyways.

Edit:

Calling this should work:

job.commit()
os._exit()