使用 Dask 连接巨大的 csv 文件

Question

I'm trying to concat three csv files (8G,4G,6G respectively) into one csv file,

and my memory is 16G, is there a way for me to concat these csv files on columns without having memory error ?

My datasets are like

A  B  C             D   E   F           G    H    I
1  2  3             4   5   6           7    8    9

My target is to merge them into

A  B  C  D  E  F  G  H  I 
  ...

My code is like

def combine_features(raw_feature_dir,connect_feature,time_feature_dir,feature_set):
df1 = dd.read_csv(raw_feature_dir)
df2 = dd.read_csv(connect_feature)
# df3 = dd.read_csv(time_feature_dir)

gc.collect()
df4 = df1.merge(df2)

df4.to_csv(feature_set)

I'm planning to merge two files first then merge the next one, but it shows memory error still

Is there a way to merge huge csv files using Dask ? or other tools

For example to compress the csv files then concat ? or to use a generator like read and write handler, that takes a chunk of data each time

Thank you!

Answer 1

I think you don't want to use merge but concat as stated in your question.

Find below a simple example :

import pandas as pd
import dask.dataframe as dd

df1 = dd.from_pandas(pd.DataFrame({'A':[1,2,3],
                                   'B':[1,4,3], 
                                   'C':[1,2,5]}), 
                                    npartitions=10)
df2 = dd.from_pandas(pd.DataFrame({'D':[0,2,3], 
                                   'E':[1,9,3], 
                                   'F':[1,6,5]}), 
                                    npartitions=10)

dd.concat([df1,df2], axis=1).head(5, npartitions=2)

Output:

   A  B  C  D  E  F
0  1  1  1  0  1  1
1  2  4  2  2  9  6
2  3  3  5  3  3  5

Answer 2

A CSV is row-like storage, so it's not easy to append whole columns. One option, as mentioned in a comment, is to split your data into smaller chunks, add columns to the chunks of your CSV, and then append that chunk to a local CSV file you are building (on disk, not in memory).

You could use skiprows and nrows options of the pandas read_csv method to read in a specific range of indices from your 3 files, combine into one dataframe in memory (representing a chunk of your desired CSV), and then append to the CSV you are building on disk.

Another option is to use a different storage format that might allow appending columns more efficiently. Dask seems to have a few options .

Dask also has a single_file option for its to_csv method, but I don't think it will help in your case since you need to append columns.

Answer 3

I will assume that you have standard csv files. The less memory consuming way is to only use the csv module. That way you will process one line at a time:

def combine_features(raw_feature_dir,connect_feature,time_feature_dir,feature_set):
    with open(raw_feature_dir) as fd1, open(connect_feature) as fd2, open(time_feature_dir) as fd3,open(feature_set, "w") as fdout:
        fds = [fd1, fd2, fd3]
        readers = [csv.reader(fdi) for fdi in fds]
        writer = csv.writer(fdout)
        try:
            while True:
                row = [field for field in r for r in [next(reader) for reader in readers]]
                writer.writerow(row)
        except StopIteration:
            pass

Beware: above code assumes that:

• all the rows in all the input csv files are correct (no row with a different number of fields than the header of same file)
• all the csv files have the same length

If those assumptions may be wrong, the code should:

• store the lenght of the first line of each file
• for each row to have same length by adding empty fields if the row if too short or truncating it if it is too long
• wait for end of the longest file instead of the shorter one.

Not shown here, because it would add a good deal of complexity while in most use cases the assumptions are reasonable...