如何计算每组中最频繁的组合

Question

I have a pandas DataFrame with post_ID and tag_ID in a long format (one post to many tags).

+---------+--------+
| post_ID | tag_ID |
+---------+--------+
|       1 |      1 |
|       1 |      2 |
|       1 |      3 |
|       2 |      1 |
|       2 |      4 |
|       2 |      6 |
|       3 |      1 |
|       4 |      5 |
|       4 |      6 |
|     ... |    ... |
+---------+--------+

My question is: when looking at tags grouped by post_ID, what are the most frequent two tag combinations? As a result, I would like to have a frame that contains results like this:

+---------------------+-----+
| tag_ID_combinations |  n  |
+---------------------+-----+
|                 1,2 |  50 |
|                 3,4 | 200 |
|                 5,6 |  20 |
+---------------------+-----+

Tags 1,2 and 3 for post_ID 1 should count as 1,2 , 1,3 and 2,3 if possible. But an aggregation like 1,2,3-1x ; 1,4,6-1x ; 1-1x and 5,6-1x would also be very helpful.

Answer 1

You could use DataFrame.groupby('col').agg(func) along with itertools.combinations to get all of the 2 tag combinations and then use collections.Counter to get the number of occurrences for each combination.

from collections import Counter
from itertools import combinations
import pandas as pd

groups = df.groupby('post_ID').agg(lambda g: list(combinations(g, 2)))
combos = pd.DataFrame(
    Counter(groups.tag_ID.sum()).items(),
    columns=['tag_ID_combos', 'count']
    )

Following example alters some of the data from your question so that there will be at least a couple of tag combinations that occur more than once.

from collections import Counter
from itertools import combinations
import pandas as pd

data = [(1,1),(1,2),(1,3),(2,1),(2,3),(2,6),(3,1),(4,3),(4,6)]
df = pd.DataFrame(data, columns=['post_ID', 'tag_ID'])
print(df)
#    post_ID  tag_ID
# 0        1       1
# 1        1       2
# 2        1       3
# 3        2       1
# 4        2       3
# 5        2       6
# 6        3       1
# 7        4       3
# 8        4       6

groups = df.groupby('post_ID').agg(lambda g: list(combinations(g, 2)))
combos = pd.DataFrame(Counter(groups.tag_ID.sum()).items(), columns=['tag_ID_combos', 'count'])
print(combos)
#   tag_ID_combos  count
# 0        (1, 2)      1
# 1        (1, 3)      2
# 2        (2, 3)      1
# 3        (1, 6)      1
# 4        (3, 6)      2

Answer 2

Here is a solution if you just want to aggregate the occurrence count by post_ID. This solution would count according to your example (post_id == 1):

[1, 2, 3]: 1

and not all possible combinations:

[1, 2] = 1, [1, 3] = 1, [2, 3] = 1

df = df.groupby('post_ID')['tag_ID'].apply(list)
df = pd.DataFrame(df).reset_index()

# only if you want to throw out single occurrences
df = df[df['tag_ID'].map(len) > 1]

# cast the sorted lists to string
df['tag_ID_AS_STRING'] = [str(sorted(x)) for x in df['tag_ID']]
result = df['tag_ID_AS_STRING'].value_counts()

Answer 3

You can use group by . You can use the following

df.groupby(['post_ID', 'tag_ID']).count()

This will generate a table with the combination as the index.

Another way is to create a combination

df['combo'] = df[['post_ID', 'tag_ID']].agg(tuple, axis=1)

Then do the group by on the combo field.

Both of the above requires more work, which I am sure you can do from the above.

Answer 4

The second kind of aggregation you mention is pretty straightforward to obtain:

df = pd.DataFrame({'post_ID': [1, 1, 1, 2, 2, 2, 3, 4, 4], 
                   'tag_ID': [1, 2, 3, 1, 4, 6, 1, 5, 6]})

df.groupby('post_ID').tag_ID.unique().value_counts()

# [1]          1
# [1, 4, 6]    1
# [1, 2, 3]    1
# [5, 6]       1
# Name: tag_ID, dtype: int64

The first kind of aggregation you asked for is inconsistent, which makes it hard to answer. For post_ID 1 you are asking for 1,2 , 1,3 and 2,3, without counting the combination of an element with itself (1,1 , 2,2, etc.). Yet for post_ID 3, you do say you want 1-1x, which is not a combination of tags. If the latter is an error, you could just do this, even if it's not very elegant:

First, get the combinations for each post_ID :

import itertools

combs_df = df.groupby('post_ID').tag_ID\
    .apply(lambda x: list(itertools.combinations(x.tolist(), 2)))

combs_df

# post_ID
# 1    [(1, 2), (1, 3), (2, 3)]
# 2    [(1, 4), (1, 6), (4, 6)]
# 3                          []
# 4                    [(5, 6)]
# Name: tag_ID, dtype: object

Now, you flatten them by putting each row's list in a list:

combs_lst = []
combs_df.apply(lambda x: combs_lst.extend(x))

combs_lst

# [(1, 2), (1, 3), (2, 3), (1, 4), (1, 6), (4, 6), (5, 6)]

Now, it's trivial just to make the list as pandas series and do a value_count :

pd.Series(combs_lst).value_counts()

# (1, 4)    1
# (5, 6)    1
# (1, 6)    1
# (4, 6)    1
# (2, 3)    1
# (1, 3)    1
# (1, 2)    1
# dtype: int64