python pipe limited to 196608 ( 65536*3 ) with subprocess.Popen()

Question

I have this function call_external_command If len(data) <= 196608 it's ok, but if len(data) > 196608 it hang for ever.

def call_external_command(command, data):
    try:
        p = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=False)
        p.stdin.write(data)
    except:
        return None
    else:
        return p.communicate()[0]

Why this restriction ? And How to work around ?

Answer 1

Oh great ! Indeed we have to use communicate() instead of stdin.write() :

def call_external_command(command, data):
    p = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=False)
    return p.communicate(data)[0]

Answer 2

As oyu found out yourself, .communicate() handles this.

The reason for your observations are the following:

• Two pipes connect your program and the subprocess, namely its stdin and its stdout.
• Depending on the OS, each of these pipes has a buffer size of 65536 bytes.
• Besides, depending on the program and what it does, it reads 65536 bytes as well and writes some data out.

So, if you write 196608 bytes, the first 65536 bytes are sent to stdin, read by the program and (if it is cat , for example) output to the stdout pipe. The second 65536 bytes are put to stdin, read by the program and tried to write to stdout, but there it blocks as stdout is full. The third 65536 bytes are written to stdin. For each excess byte, writing blocks because stdin is full as well.

The solution is as you write: let .communicate() handle the whole thing. It is prepared for this situation and handles it, depending on the OS either with threading, poll or select calls.