Python Subprocess with double quotes

Question

I've got a Docker Service Log that takes in NiFi Actions and I want to capture only Log Entries that include "Successfully sent" and "Failed to process session" (and nothing more). They should be captured in a directory called "nifi_logs" in the present working directory. I need to do all of this using Python.

This is what I got so far:

docker_log = 'docker service logs nifi | grep -e "Successfully sent" -e "Failed to process session" >> $PWD/nifi_logs/nifi1.log'
subprocess.Popen(docker_log, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

I believe subprocess.Popen() is having difficulty with the double quotes used in the grep , as nifi1.log is completely empty. If the first command looks like the following:

docker_log = 'docker service logs nifi | grep session >> $PWD/nifi_logs/nifi1.log'

The Python code works just fine and captures all log entries with "session" in nifi1.log . As I explained above though, I need to grep for 2 kinds of Log Entires and both include multiple words, meaning I need to use quotes.

If I were to just run this command on the Terminal without Python:

docker service logs nifi | grep -e "Successfully sent" -e "Failed to process session" >> $PWD/nifi_logs/nifi1.log

The log generates the entries just fine, so I know the Docker Service command is written correctly.

I've tried switching the single and double quotes around, I've tried using \" instead of " within the single quotes... nifi1.log continues to be empty.

I also tried using os.system() instead of subprocess.Popen() , but I run into the same problem (and I believe os.system() is somewhat deprecated).

Any ideas what I'd need to do to change what docker_log equals so that it will properly grep for the 2 search criteria? So you're aware: this question is not asking HOW I generate the log entries (I know what Docker Services I'm looking for, they generate properly), just what I need to do to get Python Subprocess Popen to accept a command with quotes in it.

Answer 1

Thank you for your assistance @David . Looking at your example, I found a solution: I removed stdout=subprocess.PIPE from subprocess.Popen and now it accepts double quotes just fine!

docker_log = 'docker service logs nifi | grep -e "Successfully sent" -e "Failed to process session" >> $PWD/nifi_logs/nifi1.log'
subprocess.Popen(docker_log, shell=True, stderr=subprocess.STDOUT)