在 python 中合并多个大型数据帧的最佳方法是什么？

Question

I'm working on a Google Analytics API, which pulls all of the dimensions and metrics I need and sorts them into dataframes. My code has nine dataframes in total.

When I try to merge the dataframes I keep getting a "Killed: 9" error message. I know my code is inefficient and is probably taking up a ton of memory as it churns through merge after merge but I don't know how to fix it.

Here's a sample of the merges...

MergeThree = pd.merge(MergeTwo, dfFour, how = 'outer', on = ['A', 'B', 'C', 'D']).fillna(0)
MergeThree = MergeThree[[
#dimensions
'A', 'B', 'C', 'D', 'E', 'F',
'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O',
'P',
#metrics
'Q', 'R', 'S', "T", 'U',
'V', 'W', 'X', 'Y', 'Z', 'AA', 'AB', "AC"
]]

MergeFour = pd.merge(MergeThree, dfFive, how = 'outer', on = ['A', 'B', 'C', 'D']).fillna(0)
MergeFour = MergeFour[[
#dimensions
'A', 'B', 'C', 'D', 'E', 'F',
'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O',
'P', 'AD',
#metrics
'Q', 'R', 'S', "T", 'U',
'V', 'W', 'X', 'Y', 'Z', 'AA', 'AB', "AC"
]]

MergeFive = pd.merge(MergeFour, dfSix, how = 'outer', on = ['A', 'B', 'C', 'D']).fillna(0)
MergeFive = MergeFive[[
#dimensions
'A', 'B', 'C', 'D', 'E', 'F',
'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O',
'P', 'AD', 'AE',
#metrics
'Q', 'R', 'S', "T", 'U',
'V', 'W', 'X', 'Y', 'Z', 'AA', 'AB', "AC"
]]

ect. 

I've tried many different versions of the merges and the only one I can kind of get to work looks like this..

def MergeProcessThree(x):
    MergeThree = pd.merge(x, dfFourX, how = 'outer', on = ['A', 'B', 'C', 'D']).fillna(0)
    MergeThree = MergeThree[[
    #dimensions
    'A', 'B', 'C', 'D', 'E', 'F',
    'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O',
    'P',
    #metrics
    'Q', 'R', 'S', "T", 'U',
    'V', 'W', 'X', 'Y', 'Z', 'AA', 'AB', "AC"'
    ]]
    MergeThree.to_csv('MergeThree.csv.gz', mode='a', index=False, compression='gzip')

MergeTwoX = pd.read_csv('MergeTwo.csv.gz', chunksize=100, compression='gzip')

for i in MergeTwoX:
    MergeProcessThree(i)

print('Merge Three Complete')

def MergeProcessFour(x):
    MergeFour = pd.merge(x, dfFiveX, how = 'outer', on = [''A', 'B', 'C', 'D']).fillna(0)
    MergeFour = MergeFour[[
    #dimensions
    'A', 'B', 'C', 'D', 'E', 'F',
    'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O',
    'P', 'AD',
    #metrics
    'Q', 'R', 'S', "T", 'U',
    'V', 'W', 'X', 'Y', 'Z', 'AA', 'AB', "AC"
    ]]
    MergeFour.to_csv('MergeFour.csv.gz', mode='a', index=False, compression='gzip')

MergeThreeX = pd.read_csv('MergeThree.csv.gz', chunksize=100, compression='gzip')

for i in MergeThreeX:
    MergeProcessFour(i)

print('Merge Four Complete')

etc.

But the data doesn't look right. It looks like it's essentially being doubled but things are missing that are in the normal merges that aren't in the ones broken out by chunk.

I know there has to be a better way to this and get the results I'm looking for.

Any help on this would be greatly appreciated!

Answer 1

As Chaos mentioned, there isn't a fixed way to do the compression, sometimes you could gain a lot from it and other times may not help much.

The general idea is you can use less precision to represent your number if that does not change the original value or it's within the tolerated threshold. For example if a column is for sure only binary value {0, 1}s, then you can just use np.int8 instead of common np.int32 or 64, you can do that by simply df[binary_column_name] = df[binary_column_name].astype(int) , another example, np.float16(1.23456789)=1.234 if this truncation is acceptable to your application.

You can write a function that does this somewhat automatically,

• First you check if the column is integer
  • Then check if it contains negative values
    • Positive: check the value range it falls into, eg if max is less that 2^8=256 then you know you can represent it with np.int8, if less than 2^16 then you can represent it with np.int16
    • Negative: similar as positive, but now check if your value falls in eg np.iinfo(np.int8) -> min=-128, max=127
• Float: Similar as above, check the value range and the precision you wanted

You can either look at system info, or pandas.DataFrame.memory_usage to compare how much memory reduction you get after doing above steps.

Also note, some system does not support certain dtypes, so you may need to convert it to the accepted dtypes after the merge. (eg if you want to save a df to feather, it does not accept float16 afik)