find inside Python subprocess returns 1

Question

I need to search the contents of a directory with the following GNU find command:

find path -type f -name file1 -o -name file2 -o -name file3

When I execute this command in my Linux shell, the find command returns with exit code 0. When I execute the same command within a subprocess call, the find command returns exit code 1:

import subprocess  
import shlex  
findcmd = "/depot/findutils/bin/find /remote/scratch/results -type f -name 'QUEUED' -o -name 'run.pid' -o -name 'PID'"
try:
    output = subprocess.check_output(shlex.split(findcmd))
except subprocess.CalledProcessError, cpe:
    print cpe.output
    raise cpe

Output:

Traceback (most recent call last):
  File "./getaverages.py", line 63, in <module>
    raise cpe
subprocess.CalledProcessError: Command '['/depot/findutils/bin/find', '/remote/scratch/results', '-type', 'f', '-name', 'QUEUED', '-o', '-name', 'run.pid', '-o', '-name', 'PID']' returned non-zero exit status 1

The strange part is that the CalledProcessError object output atribute has exactly the same output that I get when I run find in the Linux shell (the returned output has about 15K lines). I also tried setting bufsize=-1 but that didn't help.

Any suggestion for understanding this behavior?

I'm using Python 2.7.2 and find version is 4.2.20.

Answer 1

Despite the problem you have found, for such a simple thing you are trying to achieve, I wouldn't shell-out , use os.walk instead:

import os, os.path
search = 'file1 file2 file3'.split()
for root, dirs, files in os.walk('/path'):
  for f in filter(lambda x: x in search, files):
    # do something here
    fn = os.path.join(root, f)
    print 'FOUND', fn

Answer 2

If you use plumbum , the task would be as simple as:

from plumbum.cmd import find
cmd = find['/remote/scratch/results']['-type', 'f']['-name','QUEUED']['-o']['-name', 'run.pid']['-o']['-name', 'PID']
cmd() # run it

You don't need to worry about escaping, which I'm going to guess is the cause of your trouble.

Answer 3

It seems that in the middle of the 15K output I missed the following lines:

/depot/findutils/bin/find: /remote/scratch/results/tmp.na.8Em5fZ: No such file or directory
/depot/findutils/bin/find: /remote/scratch/results/tmp.na.k6iHJA: No such file or directory
/depot/findutils/bin/find: /remote/scratch/results/tmp.na.ZPe2TC: No such file or directory

It turns out that the path I'm searching contains simulation results and it is being deleted regularly for files older than 3 days. It seems that when the delete happens when find is being executed is the root cause of the issue.