Python / Pandas-在庞大的数据框上应用函数时出现内存问题

Question

I have a dataframe which has 350 million rows , 3 columns

Requirement :

I want to split DESCRIPTION column into LIST based on pipe symbol using less memory

input_df.head():

    startTime   DESCRIPTION                                                                                                                                     Response_Time
    1504212340  Business Transaction Performance|Business Transactions|Hexa|mBanking Confirmation.(Confirmation.aspx).no|Average Response Time (ms)_value       6
    1504212340  Business Transaction Performance|Business Transactions|Hexa|mBanking Frontpage.ci|Average Response Time (ms)_value                              4
    1504202341  Business Transaction Performance|Business Transactions|Hexa|mBanking Fonto KTList GenericNS.(GenericNS).dk|Average Response Time (ms)_value     5
    1504202341  Business Transaction Performance|Business Transactions|Hexa|mBanking Transaction Overview.co|Average Response Time (ms)_value                   5
    1504202342  Business Transaction Performance|Business Transactions|Hexa|mBanking Logon.(BidError.aspx).no|Average Response Time (ms)_value                  8

desired_output:

    startTime   list_Description                                                                                                                                             Response_Time
    1504212340  ['Business Transaction Performance', 'Business Transactions', 'Hexa', 'mBanking Confirmation.(Confirmation.aspx).no', 'Average Response Time (ms)_value']    6
    1504212340  ['Business Transaction Performance', 'Business Transactions', 'Hexa', 'mBanking Frontpage.ci', 'Average Response Time (ms)_value']                           4
    1504202341  ['Business Transaction Performance', 'Business Transactions', 'Hexa', 'mBanking Fonto KTList GenericNS.(GenericNS).dk', 'Average Response Time (ms)_value']  5
    1504202341  ['Business Transaction Performance', 'Business Transactions', 'Hexa', 'mBanking Transaction Overview.co', 'Average Response Time (ms)_value']                5
    1504202342  ['Business Transaction Performance', 'Business Transactions', 'Hexa', 'mBanking Logon.(BidError.aspx).no', 'Average Response Time (ms)_value']               8

my code:

    import pandas as pd
    import glob

    path = r'C:/Users/IBM_ADMIN/Desktop/Delete/Source/app_dynamics/*'    #500 csv files in this location
    all_files = glob.glob(path) 

    #Get the input files and concatenate   
    generator  = (pd.read_csv(f, delimiter='\t', dtype=float) for f in all_files)   #Using parentheses returns a generator instead of a list, mentioning 'dtype=float' helps to use less memory
    input_df   = pd.concat(generator , ignore_index=True)   #results in 350 million rows , 3 columns
    input_df['list_Description'] = input_df['DESCRIPTION'].str.split('|')  #Splitting the string into list

My code's Drawbacks

The above code works good for less number of rows in a dataframe. But If I apply it for 350 million rows ,my memory is hitting 98% instantly and system hangs.

A csv might have helped.. BUT

If I have the 'input_df' in a csv file then, I can process in chunks(btw, in this case I don't want to write the 'input_df' to a csv :-) ). Since the above 'input_df' is a dataframe I don't know how to start with. It would be good if there is way to use chunksize directly on dataframe

Can some one give a better idea to avoid memory issue please?

Answer 1

I can't guarantee this is going to work, since I don't have the same data on which to test it, but could you apply your splitting function on the chunks as you read them so that you don't have to hold that massive column in memory twice?

Amending your code, you could try the following:

import pandas as pd
import glob

path = r'C:/Users/IBM_ADMIN/Desktop/Delete/Source/app_dynamics/*'    #500 csv files in this location
all_files = glob.glob(path) 

def read_and_split(f):
    chunk = pd.read_csv(f, delimiter='\t', dtype=float)
    chunk['list_Description'] = chunk['DESCRIPTION'].str.split('|')
    return chunk.drop('DESCRIPTION', axis=1)

#Get the input files and concatenate   
generator  = (read_and_split(f) for f in all_files)   #Using parentheses returns a generator instead of a list, mentioning 'dtype=float' helps to use less memory
input_df   = pd.concat(generator, ignore_index=True)   #results in 350 million rows , 3 columns

If this still ends up not working, you may check out Dask , which allows you to store very large DataFrames in a distributed capacity.

Answer 2

Since it appears like the DESCRIPTION column contains a lot of duplicate values, you could build a lookup table like so:

lookup = input_df.DESCRIPTION.drop_duplicates().reset_index()
lookup = lookup.reset_index().rename(columns={'index': 'description_id'})
input_df = input_df.merge(lookup, on='DESCRIPTION')
lookup = pd.concat([lookup, lookup.DESCRIPTION.str.split('|', expand=True)],
                   axis=1)

At this point you can get rid of the DESCRIPTION columns in both the lookup and the input_df , since all the necessary information is contained within the columns of the lookup data frame.

input_df.drop('DESCRIPTION', axis=1, inplace=True)
lookup.drop('DESCRIPTION', axis=1, inplace=True)

The input_df will now have an description_id column which tells you which row of the lookup data frame contains the info extracted from the DESCRIPTION .