Python: Run bash command with redirection in background and get the process id

Question

I am using Python 2.6.6 from Centos 6 platform.

I need to run a bash command from Python in background which contains redirection operators and need to read the background process's pid from the sub-process object.

I have tried the following code snippets, but it is not working.

My Code :

import subprocess

# Below code throws child_exception
myProcess = subprocess.Popen('tail -f -n 0 /home/user123/mainFile.txt >> /home/user123/tailFile.txt &', subprocess.PIPE)

#If I use the below command, terminating the process kills
#only the shell process and leaves the tail process as orphan
myProcess = subprocess.Popen('tail -f -n 0 /home/user123/mainFile.txt >> /home/user123/tailFile.txt', shell=True, subprocess.PIPE)

cmd = ['tail', '-f', '-n', '0', '/home/user123/mainFile.txt', '>>', '/home/user123/tailFile.txt', '&']
#Below line throws bash error saying: "cannot open file '>>'"
myProcess = subprocess.Popen(cmd, stdout=subprocess.PIPE)

myProcessPid = myProcess.communicate()[0]

At the end I need to get the tail process's pid which is running in background.

Answer 1

It's not useful or wise to wrap pure shell in python.

The subprocess.Popen object has ways of doing redirection and things like that by itself rather than by relying on a shell. Here's one example.

 import subprocess
 with open("ls.output.txt", "w") as f:
     # This is the way you'd run "ls -al > ls.output.txt" in the background
     p = subprocess.Popen(["ls", "-al"], stdout=f) # This will run in the background

 p.wait() # Wait till process completes

 with open("ls.output.txt") as f:
        print (f.readline()) # Will print first line of ls -al output (total 2657828)

Answer 2

These redirections won't be performed unless shell=True is passed to your subprocess.Popen() call. The better way to do it is to use the stdout=subprocess.PIPE option and capture the output yourself.

The shell normally does the redirection for you. and this deal where you break the command up into a vector (list) of command and arguments is for the command as the shell would have passed those to an execve() system call. The shell redirection, piping, and other operators aren't part of that.

Also you don't need the & operator as your subprocess.Popen() process is automatically running in the background. It may be blocked on I/O, and you can poll and read that from the suprocess.PIPE .

More to the point you don't have to use a subprocess running the tail command at all. If you just want to follow the end of the file you can do that using the file.seek() , file.tell() , file.readline() and the os.fstat() methods.

Here's a simple class which implements tail -f semantics directly in Python:

#!/usr/bin/env python
from __future__ import print_function
import os, time

class Tail(object):
    def __init__(self, filename):
        self.fd = open(fn, 'r')  # file descriptor
        self.off = self.fd.seek(0, 2)  # end of file: skip the previous contents
        self.buff = list()
        self.last_line = 0
        self.sz = os.fstat(self.fd.fileno()).st_size

    def sleep(self):
        self.off = self.fd.tell()
        while self.sz - self.off == 0:
            self.sz = os.fstat(self.fd.fileno()).st_size
            time.sleep(0.1)

    def follow(self):
        while self.sz - self.off > 0:
            self.buff.append(self.fd.readline())
            self.off = self.fd.tell()
            self.sz = os.fstat(self.fd.fileno()).st_size

    def fetch(self):
        end = self.last_line
        self.last_line = len(self.buff)
        return '\n'.join(self.buff[end:])

... and here's some sample code using it:

if __name__ == '__main__':
    import sys

    if len(sys.argv[1:]) < 1:
        print('Must supply filename', file=sys.stderr)
        sys.exit(1)
    fn = sys.argv[1]

    tail = Tail(fn)
    while True:
        print('... sleeping ...')
        tail.sleep()
        tail.follow()
        print(tail.fetch())

... which shows one way of using it.

I wouldn't use this class. I'd change Tail.sleep() to Tail.poll() which would just immediately return a value indicating whether there's data ready at the end of the file. I'd also use the Python Standard Library: select module for the polling and sleep. Then you could maintain a list of files that you're tailing all at once. Also the endlessly growing Tail.buff would be a problem; I'd either flush it automatically after each fetch or I'd add a method to flush it.