AWS EMR perform "bootstrap" script on all the already running machines in cluster

Question

I have one EMR cluster which is running 24/7. I can't turn it off and launch the new one.

What I would like to do is to perform something like bootstrap action on the already running cluster, preferably using Python and boto or AWS CLI.

I can imagine doing this in 2 steps:

1) run the script on all the running instances (It would be nice if that would be somehow possible for example from boto)

2) adding the script to bootstrap actions for case that I'd like to resize the cluster.

So my question is: Is something like this possible using boto or at least AWS CLI? I am going through the documentation and source code on github, but I am not able to figure out how to add new "bootstrap" actions when the cluster is already running.

Answer 1

Late answer, but I'll give it a shot:

That is going to be tough.

You could install Amazon SSM Agent and use the remote command interface to launch a command on all instances. However, you will have to assign the appropriate SSM roles to the instances, which will require rebuilding the cluster AFAIK. However, any future commands will not require rebuilding.

You would then be able to use the CLI to run commands on all nodes (probably boto as well, haven't checked that).

Answer 2

bootstrap script executed once the cluster started (first time or at the beginning), however AWS provide ssh to master and other nodes there you can write shell script,install libs, packages, python program , git clone your repo etc... Hope this may be helpful. Amit