Python subprocess.Popen ：使用 shell=True 运行多个命令，但它似乎没有等待所有命令完成

Question

I'm trying to use subprocess.Popen to run a check on Kafka consumer groups and log their state, but it doesn't appear to be waiting for all the commands to run. It isn't giving me any stdout, but its also returning an exit code of 0.

prompt = ["cd", "~/path/to/kafka_2.11-2.1.0;", "pwd;", "./bin/kafka-consumer-groups.sh", 
          "--bootstrap-server", "localhost:9092", "--describe", "--group", "groupname"]

response = subprocess.run(prompt, stdout=subprocess.PIPE, stderr=subprocess.PIPE, 
                          shell=True, check=True)
print(response)

Prints:

CompletedProcess(args=['cd', '~/path/to/kafka_2.11-2.1.0;', 'pwd;', './bin/kafka-consumer-groups.sh', '--bootstrap-server', 'localhost:9092', '--describe', '--group', 'groupname'], returncode=0, stdout=b'', stderr=b'')

The pwd command was to primarily test if it would return any kind of stout, it won't be kept.

I've looked through the docs for subprocess, and I haven't see anything that suggests that it is unable to capture multiple stdouts. Also, according to the logs, the CompletedProcess is returned in less than 10ms, while running cd ~path/to/kafka_2.11-2.1.0; pwd; ./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group groupname cd ~path/to/kafka_2.11-2.1.0; pwd; ./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group groupname cd ~path/to/kafka_2.11-2.1.0; pwd; ./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group groupname takes about 10-15s on my machine.

Please note that I'm using python3.5.2

Answer 1

I found my error. cd doesn't work with subprocess, but subprocess offers the cwd named argument that accepts the path you need to run arguments. The reason it was returning a CompletedProcess so quickly is that it was successfully changing directories and then exiting the subprocess.

Sorry, I wasn't thinking about cd being the culprit until I came across this question which answered my problem too.

Answer 2

Your error is more fundamental than you think. Your code runs

sh -c 'cd'

with $0 set to the directory, $1 set to pwd; , etc; so very far from what you want. (Just cd simply switches to your home directory; then the shell exits, without doing anything with all those arguments you passed in, and Python continues back in whichever directory was current before you launched the subprocess.)

Generally, pass a single string as the first argument with shell=True , and a list of strings when you don't have a shell.

subprocess.run(r"cd foo; pwd; use shell commands to your heart\'s content; run as many processes as you like as subprocesses of your shell subprocess", shell=True)
subprocess.run(['/bin/echo', 'one', 'single', 'process', 'with', 'arguments])