从python调用需要文件的shell脚本而不将文件写入磁盘

Question

For simplification purposes lets assume the shell script is the cat command. In the shell it would be normally called like this:

$ cat /some/path/myfile.txt

Now the file will be created dynamically and writing it to disk would be a major performance hit, present security issues and increase the file management overhead in a multiuser environment. The script cannot be modified nor can read from stdin ( cat was just a working sample).

I tried this:

import os
pin, pout = os.pipe()
pin, pout = os.fdopen(pin, 'r'), os.fdopen(pout,'w')

from subprocess import Popen
pout.write("test")
pout.flush()
p = Popen('cat /proc/self/fd/%s' % pout.fileno(), shell=True)

Even if I close pout p.poll() still returns nothing showing that the command is still waiting more input from the pipe.

How can I tell the command there's no more data coming from the pipe? Is there any other approach to solve this?

Edit

In bash, if the file content is generated by a program named prog1 this would be solved in this manner:

$ cat <(prog1)

Edit 2

If you don't need the shell then the other option mentioned by Alfe is better, though the problem still exists (ie the process is not finishing after reading the file contents)

#...same as before
p = Popen(['cat', '/dev/stdin'], stdin=pout)

Answer 1

This is what I did at the end but is not nice.

Because I'm working on CentOS and the problem I'm having (I've asked about it here: Python Popen can't open bash shell in CentOS/Red Hat ) I can't use execute='/bin/bash' that's why the solution looks a little different than the question.

What I'm passing in some configuration to the process using bash process substitution and here-documents:

file_contents = '...'
p = Popen(['/bin/bash', '-c', "cat <(cat<<'EOF_CFG'\n%s\nEOF_CFG\n)" % file_contents])

I'm now considering using tmpfs for creating named pipes on it. I think that starting the script on it first and then writing to the pipe should work... If I'm successfull I'll write it here too.

Answer 2

Fiddled around a little bit, seems that closing the pipe on the one side does not generate an eof on the associated pseudo file /proc/self/fd/<#> of the child process. Did you try it with /dev/stdin and simple piping your input to the subprocess's stdin?

Answer 3

Try using a named pipe for this. On it, EOFs are properly propagated:

$ mkfifo bla
$ python
from subprocess import Popen
p = Popen('cat bla', shell=True)
with file('bla', 'w') as f:
  f.write('test')

You can create named pipes on the fly from Python as well, of course, just use os.mkfifo() .