大型文件的高效多处理/多线程

Question

I have two large datasets full of hashes I need to do stuff against:

sample1 (roughly 15gb size):

    0000002D9D62AEBE1E0E9DB6C4C4C7C16A163D2C
    00000142988AFA836117B1B572FAE4713F200567
    000001BCBC3B7C8C6E5FC59B686D3568132D218C
    000001E4975FA18878DF5C0989024327FBE1F4DF

sample2 (roughly 5gb size):

    0000002D9D62AEBE1E0E9DB6C4C4C7C16A163D2C
    00000142988AFA836117B1B572FAE4713F200567
    000001BCBC3B7C8C6E5FC59B686D3568132D218C
    000001E4975FA18878DF5C0989024327FBE1F4DF

I am currently trying to implement multiprocessing with both of these files iterating over another set of files in a directory like below:

if __name__ == '__main__':
    hash_path = glob2.glob(r'pathtohashes*.csv')
    sample1 = pd.read_csv(r'pathtosample1hashes.csv', names=['hash'])
    sample2 = pd.read_csv(r'pathtosample2hashes.csv', names=['hash'])
    for file in hash_path:
        jobs = []
        p = multiprocessing.Process(compare_function(file, sample1, sample2))
        jobs.append(p)
        p.start()

The function compares the file against both sample files and outputs to a directory.

How can I make this more efficient? I feel as though I have too many processes with the full data set in memory when I could maintain a single item in memory and just reference it but am unsure as to how to do so. Any tips on how to make this more efficient would be helpful. Thank you for your assistance.

Answer 1

You might want to look into using standard unix tools. If you are trying to find common or missing items, be aware of the comm (aka calm ) and join commands. They are purpose-built, in C, for exactly this.