Converting a shell command with $(…) for Python subprocess.Popen()

Question

I have a docker command that works from terminal and gives expected result.

Now I need to re-write this script into python. Whenever trying to run python, the following error comes up: unknown shorthand flag: 'a' in -a See 'docker logs --help'.

Below is the command. I have been trying to play with a list array but nothing seems to be working.

Original command in terminal:

docker logs $(docker ps -a -f status=exited| grep -w "dev"| awk '{print $1}')

Python equivalent:

 output = subprocess.Popen([
    "docker",
    "logs",
    "$(docker",
    "ps",
    "-a",
    "-f",
    "status=exited",
    "|",
    "grep",
    "-w",
    "'dev'",
    "|",
    "awk",
    "'{print $1}'"
])

Answer 1

In bash $(...) is run as another process and the output is substituted in its place. Everything inside is one argument from the interpreter's perspective.

You will need to run docker ps -a -f status=exited| grep -w "dev"| awk '{print $1}' docker ps -a -f status=exited| grep -w "dev"| awk '{print $1}' docker ps -a -f status=exited| grep -w "dev"| awk '{print $1}' first and use its output as an argument to docker logs ... .

However, with python you don't need pipe through grep and awk, you can use list comprehensions to do mapping and filtering:

import subprocess
containers = subprocess.Popen(["docker", "ps", "-a", "-f", "status=exited"], stdout=subprocess.PIPE)
lines = [l.decode('utf-8') for l in containers.stdout.readlines()] # Convert bytes to strings
filtered = [line for line in lines if 'dev' in line] # Keep only lines with keyword
ids = [l.split()[0] for l in filtered]               # Select first column


for i in ids:
    print(">>>> Logs for " + i)
    subprocess.call(['docker', 'logs', i])

Answer 2

Your original bash code isn't running one command -- it's running four!

• First, it runs a pipeline consisting of three commands, with the stdout of each connected to the stdin of the next:

  • ['docker', 'ps', '-a', '-f', 'status=exited']
  • ['grep', '-w', 'dev']
  • ['awk', '{print $1}']

• Second, it uses the output of the last piece of that pipeline (the awk command) to construct a final docker logs command:

  • ['docker', 'logs'] + shlex.split(awk_pipeline_output) (roughly; because you didn't properly quote, it'll also evaluate each word of that split result as a glob expression).

---

So, if you really wanted to convert that code to Python, leaving it otherwise exactly-as-written, it would look like:

from subprocess import Popen, PIPE
import shlex

p1 = Popen(['docker', 'ps', '-a', '-f', 'status=exited'], stdout=PIPE)
p2 = Popen(['grep', '-w', 'dev'], stdin=p1.stdout, stdout=PIPE)
p3 = Popen(['awk', '{print $1}'], stdin=p2.stdout, stdout=PIPE)
p1.stdout.close()
p2.stdout.close()
pipeline_output, _ = p3.communicate()

p4 = Popen(['docker', 'logs'] + shlex.split(pipeline_outlet))