python中嵌套循环的多处理

Question

I need to write some files that take inputs from two different and very large lists. The following python code works, but due to the size of the lists and other variables involved takes a long time to run:

for n,seq in enumerate(ugFA):    
    with open("locusFASTAs/"+loci[n], 'a') as outFA:
        SeqIO.write(ugSeqs[seq.id], outFA, 'fasta')
        for m,i in enumerate(wantedContigs):
            if f[m].id==seq.id:
                SeqIO.write(MergeSeqs[i], outFA, 'fasta')
            else: continue

Data structures in the above code:

• ugFA is a list
• loci is a list
• ugSeqs is a dictionary
• wantedContigs is a list
• f is a list
• MergeSeqs is a dictionary

I have attempted to parallelise the code using multiprocessing . The following code does the job , but (i) doesn't run any quicker, (ii) doesn't seem to use more than 100% CPU, and (iii) spits out the error message shown below when finished, even though it completes the tasks in the loop:

def extractContigs(ugFA, loci, ugSeqs, wantedContigs, f, MergeSeqs):
    from Bio import SeqIO
    for n,seq in enumerate(ugFA):    
        with open("locusFASTAs/"+loci[n], 'a') as outFA:
            SeqIO.write(ugSeqs[seq.id], outFA, 'fasta')
            for m,i in enumerate(wantedContigs):
                if f[m].id==seq.id:
                    SeqIO.write(MergeSeqs[i], outFA, 'fasta')
                else: continue

pool = multiprocessing.Pool(processes=p)
r = pool.map(extractContigs(ugFA, loci, ugSeqs, wantedContigs, MergeSeqs))

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: map() takes at least 3 arguments (2 given)

Is there something I have done wrong in the construction of my code? I can I properly construct it to fully utilise the expediency of the multiprocessing module?

Answer 1

The problem is

r = pool.map(extractContigs(ugFA, loci, ugSeqs, wantedContigs, MergeSeqs))

is calling the function extractContigs (in the main thread, hence the 100% CPU) and then passing the results as an argument to pool.map . The correct signature is

pool.map(func, iterable)

For this to work in your case, you would need to rewrite the extractContigs function to take only one argument. From the looks of it, you'd need to significantly refactor your code to do this. Writing to the same file simultaneously might be a concern.

Your previous version could be appropriately modified:

def writeLocus(n):
    seq = ugFa[n]  
    with open("locusFASTAs/"+loci[n], 'a') as outFA:
        SeqIO.write(ugSeqs[seq.id], outFA, 'fasta')
        for m,i in enumerate(wantedContigs):
            if f[m].id==seq.id:
                SeqIO.write(MergeSeqs[i], outFA, 'fasta')
            else: continue

pool.map(writeLocus, range(len(ugFa)))

Please make sure to verify that things aren't getting garbled in the output due to the parallel writing identical files. Ideally, it would be best to have each worker write to its own file(s), and merge afterward.