为什么使用subprocess.Popen（）调用grep要比使用subprocess.check_output（）快得多

Question

I need to extract rows of entries in a csv-like file, and I'm using grep to do it inside a python script. I noticed that when I call grep using subprocess.check_output, it takes around 5.28 seconds to finish. But when I use subprocess.Popen, it only takes 0.002 seconds. That seems to be a massive difference and I'm wondering which one I should use. Should note that I intend to process each line as a string.

Here's part of my python script.

myenv = os.environ.copy()
myenv['LC_ALL'] = 'C'
file = data_path+'/'+files[12]
start = time.time()
match = 'chr3' + "[[:space:]]"
matched_reads = subprocess.Popen(['grep', match, file], stdout=subprocess.PIPE, env=myenv)
mathced_reads = str(matched_reads).splitlines()
end = time.time()
runtime = end-start
print("Popen Grep: ", runtime)

start = time.time()
match = 'chr3' + "[[:space:]]"
matched_reads = subprocess.check_output(['grep', match, file],env=myenv)
mathced_reads = str(matched_reads).splitlines()
end = time.time()
runtime = end-start
print("Checkoutput Grep: ", runtime)

Answer 1

You will find that calling Popen does not actually execute the program and return the output, but rather it constructs a object that references the created process. In your case, you didn't call Popen.communicate which "talks" to the the process and capture its output complete. Whereas check_output does all that for you. You will find that the communicate method will take about as long, but will actually return the desired output.

For the actual demonstration with POpen , replace

matched_reads = subprocess.Popen(['grep', match, file], stdout=subprocess.PIPE, env=myenv)

with

process = subprocess.Popen(['grep', match, file], stdout=subprocess.PIPE, env=myenv)
matched_reads, stderr = process.communicate()

Which should replicate the same behaviour as check_output to have matched_reads contain the output produced by grep .