Paramiko exec_command挂在docker exec上

Question

I am using Paramiko to test docker commands from an external system (I need to do this I can't just build the container and test it locally) and the test case that I am trying to run involves starting up Apache Spark and running one of the examples, specifically SparkPi. For some reason my python script hangs on the docker exec ... command below. However, previously perform other docker execs and have not had a problem running everything manually. It only breaks when I put everything in the script.

Command :

stdin, stdout, stderr = ssh_client.exec_command(f'docker exec {spark_container_id} bash -c \'"$SPARK_HOME"/bin/spark-submit --class org.apache.spark.examples.SparkPi \
                --master spark://$(hostname):7077 "$SPARK_HOME"/examples/jars/spark-examples_2.11-2.1.1.jar {self.slices_to_calculate}\'')
print("\nstdout is:\n" + stdout.read() + "\nstderr is:\n" + stderr.read()) 

Any idea what could be causing this? And why?

Answer 1

Found out that the reason for this is because I didn't have the get_pty=True parameter for exec_command . It must be the case that by attaching a terminal to the spark-submit command the output gets printed properly. So the solution to this would be

stdin, stdout, stderr = ssh_client.exec_command(f'docker exec -t {spark_container_id} bash -c \'"$SPARK_HOME"/bin/spark-submit ...', get_pty=True)

NOTE : By using get_pty=True the stdout and stderr of the exec_command get combined.