ML docker containers local setup done. Now how to deploy in a shared cloud platform using Kubernetes?

Question

I successfully pushed a markLogic image to our internal artifactory and we have a bootstrap shell script that when executed setup a 3-node ML cluster in anyone's local machine. Here's the script.

docker run --network=ml_clustered_network -d -p 7997-8002:7997-8002 -p 9013:9013 -p 9006:9006 --hostname=ml1.local --name=ml1.local docker.repo.com/nsdat1/ml_mldba_poc:latest
docker run --network=ml_clustered_network -d -p 17997-18002:7997-8002 --hostname=ml2.local --name=ml2.local docker.repo.com/nsdat1/ml_mldba_poc:latest
docker run --network=ml_clustered_network -d -p 27997-28002:7997-8002 --hostname=ml3.local --name=ml3.local docker.repo.com/nsdat1/ml_mldba_poc:latest
sleep 2
docker exec -it ml1.local /bin/sh -c './tmp/initialize-ml.sh && sleep 5 && curl -i -X POST --data "admin-username=admin&admin-password=admin&wallet-password=admin&realm=public" http://localhost:8001/admin/v1/instance-admin'
sleep 2
docker exec -it ml2.local /bin/sh -c './tmp/initialize-ml.sh && sleep 2'
docker exec -it ml3.local /bin/sh -c './tmp/initialize-ml.sh && sleep 2'
docker exec -it ml2.local /bin/sh -c 'JOINER_CONFIG=`curl --anyauth --user admin:admin -X GET -H "Accept: application/xml" http://ml2.local:8001/admin/v1/server-config` && sleep 2 && curl --anyauth --user admin:admin -X POST -o cluster-config.zip -d "group=Default" --data-urlencode "server-config=${JOINER_CONFIG}" -H "Content-type: application/x-www-form-urlencoded" http://ml1.local:8001/admin/v1/cluster-config'
docker exec -it ml2.local /bin/sh -c 'TIMESTAMP=`curl --anyauth --user admin:admin -X POST -H "Content-type: application/zip" --data-binary @./cluster-config.zip http://ml2.local:8001/admin/v1/cluster-config`'
docker exec -it ml3.local /bin/sh -c 'JOINER_CONFIG=`curl --anyauth --user admin:admin -X GET -H "Accept: application/xml" http://ml3.local:8001/admin/v1/server-config` && sleep 2 && curl --anyauth --user admin:admin -X POST -o cluster-config.zip -d "group=Default" --data-urlencode "server-config=${JOINER_CONFIG}" -H "Content-type: application/x-www-form-urlencoded" http://ml1.local:8001/admin/v1/cluster-config'
docker exec -it ml3.local /bin/sh -c 'TIMESTAMP=`curl --anyauth --user admin:admin -X POST -H "Content-type: application/zip" --data-binary @./cluster-config.zip http://ml3.local:8001/admin/v1/cluster-config`'
## To set up appservers, forests, databases etc.
docker exec -it ml1.local /bin/sh -c './tmp/post-setup-bootstrap.sh'

Now I'm familiar with docker but not with Kubernetes . My next step would be to deploy and manage this cluster in a shared platform for which I need to get familiar with kubernetes first and setup ML cluster in my local using Kubernetes.

Question.

1. Do I need VM like minikube ? I have docker for desktop setup with both docker and kubernetes running. Running kubectl gives below output -

    kubectl cluster-info Kubernetes master is running at https://localhost:6443

2. Could you also explain steps in brief that I need to follow ? I'd like to avoid having a VM in my local.

Answer 1

1. Docker Desktop with Kubernetes enabled is enough for developing your setup (in this setup Docker already runs in a VM).
2. Usually you'll start writing some Kubernetes YAML manifests and applying them to your cluster. In your case with the database requiring a consistent naming of nodes, etc. a StatefulSet might be the right option. There is an article by Marklogic describing this.