Running node manager inside a docker container and adding to a existing hadoop cluster

Question

I have created a hadoop cluster using ambari, now on a new vm i need to create a docker which needs to join this hadoop cluster and the docker container should be running the nodemanager.

Answer 1

Docker shouldn't run a Nodemanager. That would effectively cause a memory constrained environment to be responsible for further memory constrained JVM containers.

A Nodemanager should be installed directly on the host OS. Then YARN can be configured to run Docker containers - https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/DockerContainers.html

Alternatively, YuniKorn just became a top level Apache Project - https://yunikorn.apache.org/

(Ambari is dead, Kube.netes is the current future for data analytics cluster configuration, installation, and application deployments)