Automatically Generate Pods in Kubernetes

Question

I previously created a Flask server that spawns Docker containers using the Docker Python SDK. When a client hits a specific endpoint, the server would generate a container. It would maintain queues, and it would be able to kill containers that didn't respond to requests.

I want to migrate towards Kubernetes, but I am starting to think my current server won't be able to "spawn" jobs as pods automatically like in docker.

docker.from_env().containers.run('alpine', 'echo hello world')  

Is Docker Swarm a better solution for this, or is there a hidden practice that is done in Kubernetes? Would the Kubernetes Python API be a logical solution for automatically generating pods and jobs, where the Flask server is a pod that manages other pods within the cluster?

Answer 1

'Kubectl run' is much like 'docker run' in that it will create a Pod with a container based on a docker image (eg How do i run curl command from within a Kubernetes pod ). See https://kubernetes.io/docs/reference/kubectl/docker-cli-to-kubectl/ for more comparison. But what you run with k8s are Pods/Jobs that contain containers rather than running containers directly so this will add an extra layer of complexity for you.

Kubernetes is more about orchestrating services rather than running short-lived jobs. It has some features and can be used to run jobs but that isn't its central focus. If you're going in that direction you may want to look at knative (and knative build) or kubeless as what you describe sounds rather like the serverless concept. Or if you are thinking more about Jobs then perhaps brigade ( https://brigade.sh ). (For more see https://www.quora.com/Is-Kubernetes-suited-for-long-running-batch-jobs ) If you are looking to run web app workloads that serve requests then note that you don't need to kill containers that fail to respond on k8s as k8s will monitor and restart them for you.

I don't know swarm well enough to compare. I suspect it would be a bit easier for you as it is aimed more centrally at docker (the k8s API is intended to support other runtimes) but perhaps somebody else can comment on that. Whether using swarm instead helps you will I guess depend on your motivations.