如何使用多处理与python熊猫合并两个csv文件

Question

I want to merge two csv files with common column using python panda With 32 bit processor after 2 gb memory it will throw memory error how can i do the same with multi processing or any other methods

import gc
import pandas as pd
csv1_chunk = pd.read_csv('/home/subin/Desktop/a.txt',dtype=str, iterator=True, chunksize=1000)
csv1 = pd.concat(csv1_chunk, axis=1, ignore_index=True)
csv2_chunk = pd.read_csv('/home/subin/Desktop/b.txt',dtype=str, iterator=True, chunksize=1000)
csv2 = pd.concat(csv2_chunk, axis=1, ignore_index=True)
new_df = csv1[csv1["PROFILE_MSISDN"].isin(csv2["L_MSISDN"])]
new_df.to_csv("/home/subin/Desktop/apyb.txt", index=False)
gc.collect()

please help me to fix this

thanks in advance

Answer 1

I think you only need one column from your second file (actually, only unique elements from this column are needed), so there is no need to load the whole data frame.

import pandas as pd

csv2 = pd.read_csv('/home/subin/Desktop/b.txt', usecols=['L_MSISDN'])
unique_msidns = set(csv2['L_MSISDN'])

If this still gives a memory error, try doing this in chunks:

chunk_reader = pd.read_csv('/home/subin/Desktop/b.txt', usecols=['L_MSISDN'], chunksize=1000)
unique_msidns = set()
for chunk in chunk_reader:
    unique_msidns = unique_msidns | set(chunk['L_MSIDNS'])

Now, we can deal with the first data frame.

chunk_reader = pd.read_csv('/home/subin/Desktop/a.txt', chunksize=1000)
for chunk in chunk_reader:
    bool_idx = chunk['PROFILE_MSISDN'].isin(unique_msidns)
    # *append* selected lines from every chunk to a file (mode='a')
    # col names are not written
    chunk[bool_idx].to_csv('output_file', header=False, index=False, mode='a')

If you need column names to be written into the output file, you can do it with the first chunk (I've skipped it for code clarity).

I believe it's safe (and probably faster) to increase chunksize .

I didn't test this code, so be sure to double check it.