加入两个巨大的文件，而不会与熊猫分块

Question

I have File1 with "id,name" and File2 with "id,address". I cannot load the first file (less than 2Gb): it crashes after 76k rows (with chunk concat) and only 2 columns... I cannot read_csv on the second file too because it crashes the kernel after some rows loading.

I need to join the File1 and File2 with "id" but if I cannot put files in a dataframe variable I don't know how to do...

The file is only 5Gb with 30M rows but it crashes the kernel after few seconds of loading.

How to join the file without dataframing em please ?

I have tried with chucking but it crashes.

chunks = []
cols = [...]
for chunk in pd.read_csv("file2.csv", chunksize=500000, sep=',', error_bad_lines=False, low_memory=False, usecols=cols):
    chunks.append(chunk)
df = pd.concat(chunks, axis=0)
print(f.shape)

I need the dataframe to load to join them or join the file without loading if possible

Answer 1

You read df2 chunk by chunk but since you append all the chunks, your resulting chunk is of the same size as your file2.

What you could do, if you are able to fully load your df1, is to join your df2 chunk by chunk like so :

for chunk in pd.read_csv("file2.csv", chunksize=500000, sep=',', error_bad_lines=False, low_memory=False, usecols=cols):
    df1.merge(chunk, on =['id'], how='left')

Answer 2

Chunking like that will definitely still crash your kernel, since you're still trying to fit everything into memory. You need to do something to your chunks to reduce their size.

For instance, you could read both files in chunks, join each chunk, output the matches to another file, and keep the un-matched IDs in memory. That might still crash your kernel if you get unlucky though. It depends on what your performance constraints are, and what you need to do with your data afterwards.