在熊猫中平均不同水平

Question

I have a large dataset of music tagging data in a MySQL database that I am attempting to analyze with pandas. I exported it to .tsv from MySQL and now reading it in as a dataframe for analysis.

Each row in the data is a tuple indicating that a given user (indicated by numeric user ID) tagged a particular artist with a particular tag (represented here as a numeric ID) at a particular time. So with no indexes a sample of the data would look like this:

       uid  artist   tag        date
0  2096963     559    46  2005-07-01
1  2096963     584  1053  2005-07-01
2  2096963     584  2044  2005-07-01
3  2096963     584  2713  2005-07-01
4  2096963     596   236  2005-07-01
...
       uid  artist   tag        date
99995  2656262    8095    57  2005-08-01
99996  2656262    8095    79  2005-08-01
99997  2656262    8095  4049  2005-08-01
99998  2656262    8095  8290  2005-08-01
99999  2610168    8095  1054  2005-08-01

To facilitate analyses, I've indexed everything and added a dummy annotations variable (each row in the data represents one tagging instance, or annotation). So now we have:

data = pd.read_table(filename,header=None, names=('uid','artist','tag','date'), index_col=['date','uid','artist','tag'], parse_dates='date') 
data['annotations'] = 1

In [41]: data.head()
Out[41]:
                                annotations
date       uid     artist tag
2005-07-01 2096963 559    46              1
                   584    1053            1
                          2044            1
                          2713            1
                   596    236             1
...

With the data formatted like this, it's trivial to calculate simple frequency distributions. For instance, if I want to determine the number of times each user tagged something (in descending freq. order), it's as simple as:

data.sum(level='uid').sort('anno',ascending=False)

Similarly, I can determine the total number of annotations each month (across all users and tags) with:

data.sum(level='date')

But I'm having trouble with more complex calculations. In particular, what if I want the the mean number of annotations per user each month? If I call:

data.sum(level=['date','uid']).head()

I get the number of annotations per user each month, ie:

                    anno
date       uid
2005-07-01 1040740    10
           1067454    23
           2096963   136
           2115894     1
           2163842     4
...

but what's a simple way to then get a monthly average of those values across users? That is, for each month, what is the average across users of the the "anno" column? I have various metrics like this I want to calculate, so I'm hoping the solution generalizes.

Answer 1

Big MultiIndexes can be a hassle. I suggest abandoning your dummy column, 'annotations', and using count instead of sum .

To start, read in the data without assigning an index, ie,

pd.read_table(filename,header=None, names=['uid','artist','tag','date'], parse_dates='date')

To count each user's annotations:

data.groupby('uid').count().sort(ascending=False)

To total annotations per day:

data.groupby('date').count()

The count unique users each day:

daily_users = data.groupby('date').uid.nunique()

To total annotations each day:

daily_annotations = data.groupby('date').count()

The average daily annotations per user is just the daily total annotations divided by the number of users that day. As a result of the groupby operation, both of these Series are indexed by date, so they will align automatically.

mean_daily_annotations_per_user = daily_annotations/daily_users

To average annotations per month across users , it is most convenient to use resample , a nice feature for grouping by different time frequencies.

mean_monthly_annotations_per_user = mean_daily_anootations_per_user.resample('M')

Answer 2

I figured out an alternative approach that fits my original multi-index format, and I think is faster than the method proposed by @DanAllan.

Recalling that we're calculating the mean annotations per user per month, let's build two dataframes (I'm using just a subset of the data here, hence the nrows argument). data1 is the multi-index version with dummy variable, and data2 is the unindexed version proposed by @DanAllan

indexes=['date','uid','artist','iid','tag']
data1 = pd.read_table(filename,header=None, nrows=1000000, names=('uid','iid','artist','tag','date'),index_col=indexes, parse_dates='date') 
data['anno']=1
data2 = pd.read_table(filename,header=None, nrows=1000000, names=('uid','iid','artist','tag','date'), parse_dates='date') 

With the unindexed (data2) version the process is:

daily_users = data2.groupby('date').uid.nunique()
daily_annotations = data2.groupby('date').count().uid
anno_per_user_perday2 = daily_annotations / daily_users.map(float)

With the multi-index version (data1), we can do:

anno_per_user_perday = data1.sum(level=['date','uid']).mean(level='date').anno

The result is exactly the same, but more than twice as fast with the indexed version (performance will be more of an issue with the full, 50 million row dataset):

%timeit -n100 daily_users = data2.groupby('date').uid.nunique() ; daily_annotations = data2.groupby('date').count().uid ; anno_per_user_perday2 = daily_annotations / daily_users.map(float)
100 loops, best of 3: 387 ms per loop

%timeit -n100 anno_per_user_perday1 = data1.sum(level=['date','uid']).mean(level='date').anno
100 loops, best of 3: 149 ms per loop

Generating the dataframe is slower with the indexed version, but the flexibility it affords seems worth it.