使用熊猫进行缓慢的数据分析

Question

I am using a mixture of both lists and pandas dataframes to accomplish a clean and merge of csv data. The following is a snippet from my code that runs disgustingly slow... Generates a csv with about 3MM lines of data.

UniqueAPI = Uniquify(API)
dummydata = []
#bridge the gaps in the data with zeros
for i in range(0,len(UniqueAPI)):
    DateList = []
    DaysList = []
    PDaysList = []
    OperatorList = []
    OGOnumList = []
    CountyList = []
    MunicipalityList = []
    LatitudeList = []
    LongitudeList = []
    UnconventionalList = []
    ConfigurationList = []
    HomeUseList = []
    ReportingPeriodList = []
    RecordSourceList = []

    for j in range(0,len(API)):
        if UniqueAPI[i] == API[j]:
            #print(str(ProdDate[j]))
            DateList.append(ProdDate[j])
            DaysList = Days[j]
            OperatorList = Operator[j]
            OGOnumList = OGOnum[j]
            CountyList = County[j]
            MunicipalityList = Municipality[j]
            LatitudeList = Latitude[j]
            LongitudeList = Longitude[j]
            UnconventionalList = Unconventional[j]
            ConfigurationList = Configuration[j]
            HomeUseList = HomeUse[j]
            ReportingPeriodList = ReportingPeriod[j]
            RecordSourceList = RecordSource[j]

    df = pd.DataFrame(DateList, columns = ['Date'])
    df['Date'] = pd.to_datetime(df['Date'])
    minDate = df.min()
    maxDate = df.max()

    Years = int((maxDate - minDate)/np.timedelta64(1,'Y'))
    Months = int(round((maxDate - minDate)/np.timedelta64(1,'M')))
    finalMonths = Months - Years*12 + 1
    Y,x = str(minDate).split("-",1)
    x,Y = str(Y).split("   ",1)
    for k in range(0,Years + 1):

        if k == Years:
            ender = int(finalMonths + 1)
        else:
            ender = int(13)

        full_df = pd.DataFrame()
        if k > 0:
            del full_df
            full_df = pd.DataFrame()

        full_df['API'] = UniqueAPI[i]
        full_df['Production Month'] =     [pd.to_datetime(str(x)+'/1/'+str(int(Y)+k)) for x in range(1,ender)]
        full_df['Days'] = DaysList
        full_df['Operator'] = OperatorList
        full_df['OGO_NUM'] = OGOnumList
        full_df['County'] = CountyList
        full_df['Municipality'] = MunicipalityList
        full_df['Latitude'] = LatitudeList
        full_df['Longitude'] = LongitudeList
        full_df['Unconventional'] = UnconventionalList
        full_df['Well_Configuration'] = ConfigurationList
        full_df['Home_Use'] = HomeUseList
        full_df['Reporting_Period'] = ReportingPeriodList
        full_df['Record_Source'] = RecordSourceList
        dummydata.append(full_df)

full_df = pd.concat(dummydata)                                   
result =  full_df.merge(dataClean,how='left').fillna(0)

print(result[:100])

result.to_csv(ResultPath, index_label=False, index=False)

This snippet of code has been running for hours the output should have ~3MM lines there has to be a faster way using pandas to accomplish the goal of which I will describe:

• for each unique API i find all occurrences in the main list of apis
• using that information i build a list of dates
• I find a min and max date for each list corresponding to an api
• I then build an empty pandas DataFrame that has every month between the two dates for the associated api
• I then append this data frame to a list "dummydata" and loop to the next api
• taking this dummy data list I then concatenate it into a DataFrame
• this DataFrame is then merged with another dataframe with cleaned data
• end result is a csv that has 0 value for dates that did not exist but should between the max and min dates for each corresponding API in the original unclean list

This all takes way longer than I would expect I would have thought that finding the min max date for each unique item and interpolating monthly between them filling in months that dont have data with 0 would be like a three line thing in Pandas. Any options that you guys think I should explore or any snippets of code that could help me out is much appreciated!

Answer 1

You could start by cleaning up the code a bit. These lines don't seem to have any effect or functional purpose since full_df was just created and is already an empty dataframe:

if k > 0:
    del full_df
    full_df = pd.DataFrame()

Then when you actually build up your full_df it's better to do it all at once rather than one column at a time. So try something like this:

full_df = pd.concat([UniqueAPI[i],
                     [pd.to_datetime(str(x)+'/1/'+str(int(Y)+k)) for x in range(1,ender)],
                     DaysList,
                     etc...
                     ],
                    axis=1)

Then you would need to add the column labels which you could also do all at once (in the same order as your lists in the concat() call).

full_df.columns = ['API', 'Production Month', 'Days', etc.]