在Python中组合异类csv文件的最快/最有效方法

Question

I have 1000 csv files with size varying from 8MB to 17MB. Each file has a portion of 6 metrics. Examples are below:

File1 (columns): key, metric1, metric2, metric3
File1 (values):  k1, m1, m2, m3
File2 (columns): key, metric4, metric5, metric6
File2 (values):  k1, m4, m5, m6
File3 (columns): key, metric2, metric4, metric5, metric6
File3 (values):  k2, m2, m4, m5, m6

All methods I tried today combined files into the output below:

Output (columns): key, metric1, metric2, metric3, metric4, metric5, metric6
Output (values):  key1, m1,   m2,   m3,   null, null, null
                  key1, null, null, null, m4,   m5,   m6
                  key2, null, m2,   null, m4,   m5,   m6

What I really need is also consolidate rows by key column:

Output (columns): key, metric1, metric2, metric3, metric4, metric5, metric6
Output (values):  key1, m1,   m2, m3,   m4, m5, m6
                  key2, null, m2, null, m4, m5, m6

I know pandas could do it; however, it may take forever to finish 1000 files.

Answer 1

It's a bit unclear what your format is, but I think this will work:

 df = pd.DataFrame(expected_metrics)
 for filename in filelist:
       current_data = pd.read_csv(filename,index_col = 'key')
       current_columns = current_data.columns
       current_row = current_data.index[0] 
       df.loc[current_row,current_columns] = current_data

Notes:

-This requires you know ahead of time what metrics will be present so you can initialize expected_metrics . You could instead replace the last line with:

 for column in current_columns:
      df.loc[current_row,column] = current_data[column]

This would probably take longer

-If a particular (key, metric) combination shows up more than once, only the last one will be recorded.

-The result will have key as the index. If you want it as a data column, you'll have to do df['key'] = df.index .

I wouldn't expect this to take "forever"; a thousand files should take at worst a few minutes unless you have a very large number of metrics.

You could also do:

 data = {}
 for filename in filelist:
       current_data = pd.read_csv(filename,index_col = 'key')
       current_columns = current_data.columns
       current_row = current_data.index[0] 
       data[current_row] = {column:list(current_data[column])[0] for column in current_columns}

This will give a dictionary where each key is a key from your data, and the value is a dictionary representing the row for that key.

EDIT: A third option would be to take the output you already have and do df.groupby(by='key').max() This will create a dataframe where each entry is the maximum metric for all the rows with the same key. So, again, if you have only one value for each (key,metric) combination, this should give you what you want. If you have more than one value, all but the largest will be ignored.