subprocess breaking pipes and file busy

Question

I am trying to do some file system stuff using python, since I don't want to deal with complex shell scripting and would rather confine all my programming to python insofar as possible. The shell command in 'search_string' reads filenames in a directory and writes the first 10 names into a file .

search_string = "find " + rootDir + "/"+str(k)  +" -iname \"*\" -type f | head -10  >> negatives" + str(i) + ".txt"
print(search_string)
subprocess.call(search_string, shell=True)

This code works on my ubuntu 14.04 pc but not on aws which is where this has to run eventually, giving the error:

find: `standard output': Broken pipe
find: write error

ubuntu@ip:$ uname -r
3.13.0-37-generic

so I decided to write the long shell command to a file which i figured would be easy to call (after making the shell script file executable with the chmod command):

search_string = "sudo find " + rootDir + "/"+str(k)  +" -iname \"*\" -type f | head -10  >> trainingfiles/negatives" + str(i) + ".txt"
f=open("cmd.sh","w")
f.write(search_string)
f.flush()
os.fsync(f.fileno())
f.close
p1=subprocess.Popen(["chmod","+x","/home/www-data/web2py/applications/fingerPrint/modules/classifier_stuff/cmd.sh"],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
stdout, stderr = p1.communicate()
output = p1.communicate()[0]
print('stdout:'+stdout+' stderr:'+stderr)
sys.stdout.flush()

p2=subprocess.Popen(["sudo /home/www-data/web2py/applications/fingerPrint/modules/classifier_stuff/cmd.sh"],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
stdout, stderr = p2.communicate()
print('stdout:'+stdout+' stderr:'+stderr)
sys.stdout.flush()

but I get

stdout: stderr:
Traceback (most recent call last):
File "prepare_and_train.py", line 56, in <module>
 p2=subprocess.Popen(["/home/www-data/web2py/applications/fingerPrint/modules/classifier_stuff/cmd.sh"],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 26] Text file busy

If I change PIPE to STDOUT I get an interesting

OSError: [Errno 9] Bad file descriptor

and the same 'file busy' when I try subprocess.call :

sudo: unable to execute ./cmd.sh: Text file busy
stdout: stderr:

I don't really care how I do it, I just want working code - any tips here? I am (prob. obviously) pretty new to linux

Answer 1

The error you are encoutering is happening because you are trying to execute the script while it's still open for writing. In particular, see the following minimal example:

#!/usr/bin/env python

import os

f = open('f.sh', 'w')
f.write("#!/bin/sh\necho test")
os.chmod('f.sh', 0o755)
os.execl('f.sh', './f.sh')

If you execute it, you'll get:

$ ./a.py 
Traceback (most recent call last):
  File "./a.py", line 8, in <module>
    os.execl('f.sh', './f.sh')
  File "/usr/lib64/python3.4/os.py", line 491, in execl
    execv(file, args)
OSError: [Errno 26] Text file busy

If you ensure to close the file before executing, eg:

#!/usr/bin/env python

import os

with open('f.sh', 'w') as f:
    f.write("#!/bin/sh\necho test")
os.chmod('f.sh', 0o755)
os.execl('f.sh', './f.sh')

it works fine:

$ ./a.py 
test

---

Getting into your specific issue, it's line 6:

f.close

You are missing the parentheses, so instead of calling the close() method, you are simply obtaining (and not using) it. That should be:

f.close()

or prefably a with statement as in my example.

While at it, you may also want to use os.chmod() instead of calling external chmod tool.

Answer 2

The "broken pipe" error is normal and expected for anything which pipes to head or a similar utility.

Breaking things down a bit,

1. Process one starts writing to the pipe
2. Process two starts reading from the pipe
3. Process two finishes and closes the pipe
4. Process one tries to write more data to the pipe

When you don't have a lot of output, (4) never happens, so you don't see this error all the time. But it is normal and expected.

I would handle the head part in Python itself, anyway. Then you can do away with the shell=True , too.

find = subprocess.Popen(['find', rootDir + '/' + str(k), '-type', 'f'],
    stdout=subprocess.PIPE)
result = [find.stdout.readline() for x in xrange(10)]
find.terminate()
with open('negatives' + str(i) + 'txt', 'w') as output:
    output.write(''.join(result))

(Your -iname "*" argument to find doesn't seem to serve any useful purpose, so I left it out.)

Ironically, this is somewhat more opaque in Python than in "complex" shell script. Of course, in pure Python, you would use os.walk() instead of find .