提高熊猫python的性能

Question

Here is the example of my data set from some public transports: data set .

Date is from 2018-06-01 to 2018-06-30,

Time is operation hours, from 5am to 24(0)am,

People is the number of peope in that specific date, time and trip.

from_to is the where those people enter and leave (one type of trips),

and finally weekday.

What I need to do here is to create a time table for each trip, for example if I want to create a table for trip "G1_G2", the code I use now is:

for i in [0,1,2,3,4,5,6]:
    for j in [0,1,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23]:
        df['people'][(df['weekday'] == i)&(df['from_to'] == 'G1_G2') & (df['time'] == j)].mean()

Where "i" is weekday, and j is the operating hours. The result will be a table like: output table

But the problem here is that each table will take about 10 seconds to create, there are around 11,000 types of trip here, which will take 30 something hours.

Is there other ways to do this with higher efficiency using python?

Thanks in adanvance!

Answer 1

Probably using groupby and aggregate you can do this.

import pandas as pd

Deliberately I use here a small data example. If you have many smaller tables as I interpret from your description, you may want to concatenate before.

df = pd.DataFrame({'date':['2018-06-01', '2018-06-01', '2018-06-01', '2018-06-02', '2018-06-02', '2018-06-02'], 'time':[0,0,0,1,1,1], 'people':[0,2,2,4,5,7], 'from_to':['BR13_BR13', 'BR13_BR13','BR13_BR13','BR13_BR13','BR13_BR13','BR13_BR13'], 'weekday':[4,4,4,5,5,5]})

The following code results in a long format, not in the wide format that your output table has, but it can be made wide if you wanted:

df.groupby(['from_to', 'time', 'weekday']).aggregate('people').mean()

Which results in the following output:

from_to    time  weekday
BR13_BR13  0     4          1.333333
           1     5          5.333333
Name: people, dtype: float64