如何添加所有键的值并打印新词典？

Question

I have a file1 has region information like chromosome1 from position 1 to position 10, looks like: chromosome,start_position,end_position 1,1,10 1,11,20

A file2 has values for every position like position 6 on chromosome1 with some value, looks like: chromosome,position,value 1,1,value1 1,2,value2 1,6,value3 1,13,value4

I want to add values in file2 to file1, based on whether positions in file2 belongs to any region in file1 ,something like: chromosome,start_position,end_position,total_value 1,1,10,value1+value2+value3 1,11,20,value4

Both files can be more than 10 million lines, Should I do this by looking through every line of file2 (to see if the position be in any region of file1), or making every line of file1 a dictionary (then find value in file2? then add?)?

And how can I get the 'total value' of every line in file1? Thanks everyone!

Answer 1

I'm presuming that you're not necessarily looking for the most efficient code, but one that gets the job done?

I would read the values in file 2 into a dictionary, with the key being a (chromosome, start) pair (presuming that the start and end are always the same in file 2).

Then read file 1 line-by-line, and find all relevant values in your "file 2" dictionary, appending the resultant sum to the end of the line (probably in a new file):

import numpy as np

for line in file1:
    chromosome, start, end = line.split(',')
    total_value = np.sum([file2_dict.get([(chromosome,str(i))], 0) for i in
                          range(int(start), int(end)+1)])
    #do something with the total value, maybe write to another file.
    #Could do:
    new_line = ','.join([chromosome, start, end, total_value]) + '\n'

I'm going to leave the rest of the implementation details to you (such as getting your dictionary from file 2). It might be a bit heavy on memory usage, but hopefully not too bad.

Note the use of the .get() method with the dictionary lookup - this will make sure that any key that isn't found in the dictionary returns 0 . You decide if this works for your situation. Also note the use of str and int to convert between text and numbers. You decide if this is appropriate based on your implementation.

Also, if you haven't come across Python list comprehensions before, do some research on that. That is what allows us to write the one-liner to get the sum of all relevant values.