Python多处理未使用完整的CPU核心

Question

I used regex to check sequence record of a paired-end fastq files and write the matched sequences into the same files. I used multiprocessing to speed up it but when I ran it with 20 processes, the 20 cpu cores were all using 2% and the total time was the same as running in a single core. Does it mean the regex search is faster than writing output to file so the processes were waiting? Can you suggest how can I improve the multiprocessing? Attached is the code.

def mycallback(x):
    SeqIO.write(x[0],outfile1,result.form)
    SeqIO.write(x[1],outfile2,result.form)
    SeqIO.write(x[2],outfile3,result.form)
    SeqIO.write(x[3],outfile4,result.form)

def check(x):
    if regex.search(r'^.{0,20}(?:'+fp+'){e<='+str(result.mm)+'}',str(x[0].seq),flags=regex.I) and regex.search(r'^.{0,20}(?:'+rp+'){e<='+str(result.mm)+'}',str(x[1].seq),flags=regex.I):
    return((x[0],x[1],'',''))
    elif regex.search(r'^.{0,20}(?:'+fp+'){e<='+str(result.mm)+'}',str(x[1].seq),flags=regex.I) and regex.search(r'^.{0,20}(?:'+rp+'){e<='+str(result.mm)+'}',str(x[0].seq),flags=regex.I):
    return((x[1],x[0],'',''))
    else:
    return(('','',x[0],x[1]))

p=Pool(int(result.n))
for i in izip(SeqIO.parse(result.fseq,result.form),SeqIO.parse(result.rseq,result.form)):
    p.apply_async(check,args=(i,),callback=mycallback)

p.close()
p.join()

Answer 1

Python's implementation of pool.apply_async calls the callback function inside a thread inside the main process and is such limited by the GIL . You are thus waiting on all your file writes sequentially.

Callbacks should complete immediately since otherwise the thread which handles the results will get blocked.

I would imagine your regex executes faster than file writing, so you would benefit the most from sending the callbacks to their own threads (so multiple file writes can be queued at once). Python threads should release the GIL when waiting on IO (file writes), and are much lighter (faster to start up) than processes.