在 EMR 中取消 YARN 步骤

Question

I have a long running YARN application running on EMR cluster. Based on Canceling EMR Steps , the running steps can be canceled with command aws emr cancel-steps as long as Amazon EMR versions 5.28.0 and later is being used (which is the case for me), however when I issue the above against my running step it never kills the actual yarn application. I can see the step changing it's status to Canceled in the UI, however if I ssh into the EMR and execute yarn application -list I still can swe my application alive and well:) In the logs I see

INFO waitProcessCompletion ended with exit code 137 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 344 seconds
2020-12-30T23:13:42.362Z INFO Step created jobs: 
2020-12-30T23:13:42.362Z WARN Step failed with exitCode 137 and took 344 seconds

Which, based on my understanding, means that the container did receive the SIGKILL command. Can someone advise why it is still not killing the application?

PS I am using the TERMINATE_PROCESS cancelation option when executing the cancel-steps command.

Thank you!

Answer 1

One aspect that you might want to look at is the cluster vs client mode of submitting the step. If you are using cluster mode, you can't kill that step through AWS SDK since it has no control of the actual yarn application, for this to work you need to run your steps in the client mode.

Please also check other implications of running in cluster vs client mode, if you are doing this for development then it is fine to use client mode.