Django google kubernetes client not running exe inside the job

Question

. 。 Answers to this question are eligible for a +50 reputation bounty. Turgut wants to to this question: 这个问题：

To provide a solution for my situation.

I have a docker image that I want to run inside my django code. Inside that image there is an executable that I have written using c++ that writes it's output to google cloud storage. Normally when I run the django code like this:

container = client.V1Container(name=container_name, command=["//usr//bin//sleep"], args=["3600"], image=container_image, env=env_list, security_context=security)

And manually go inside the container to run this:

gcloud container clusters get-credentials my-cluster --region us-central1 --project proj_name  && kubectl exec pod-id -c jobcontainer -- xvfb-run -a "path/to/exe"

It works as intended and gives off the output to cloud storage. (I need to use a virtual monitor so I'm using xvfb first). However I must call this through django like this:

container = client.V1Container(name=container_name, command=["xvfb-run"], args=["-a","\"path/to/exe\""], image=container_image, env=env_list, security_context=security)

But when I do this, the job gets created but never finishes and does not give off an output to the storage. When I go inside my container to run ps aux I get this output:

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0   2888  1836 ?        Ss   07:34   0:00 /bin/sh /usr/bin/xvfb-run -a "path/to/exe"
root          16  0.0  1.6 196196 66256 ?        S    07:34   0:00 Xvfb :99 -screen 0 1280x1024x24 -nolisten tcp -auth /tmp/xvfb-run.r5gaBO/Xauthority
root          35  0.0  0.0   7016  1552 ?        Rs   10:31   0:00 ps aux

It looks like it's stuck inside my code but my code does not have a loop that it can stuck inside, perhaps there is an error occurring (I don't think so since the exact same command is working when typed manually). If there is an error how can I see the console output? Why is my code get stuck and how can I get my desired output? Could there be an error caused by permissions (The code does a lot of stuff that requires permissions like writing to storage and reading files inside the pod, but like mentioned works normally when i run it via the command line)?

Answer 1

Apparently for anyone having a similar issue, we fixed it by adding the command we want to run at the end of the Dockerfile instead of passing it as a parameter inside django's container call like this:

cmd["entrypoint.sh"]

entrypoint.sh:

xvfb-run -a "path/to/exe"

Instead of calling it inside django like we did before and simply removing the command argument from the container call so it looked like this:

container = client.V1Container(name=container_name, image=container_image, env=env_list, stdin=True, security_context=security)

Answer 2

When working with Docker and Django it's quite common to get things into a state where the container keeps restarting / won't really start. This usually indicates a problem in Django and the logs shows us what's wrong with it (no need to guess).

So, one has to check the container logs

docker logs CONTAINER

That is going to give one more details about the error and, from there, one will know how to proceed to fix it since one will get a more specific error.