Recommend based on search key word

Question

I have a input query table in the following:

    query
0  orange
1   apple
2    meat

which I want to make against the user query table as following

   user       query
0    a1      orange
1    a1  strawberry
2    a1        pear
3    a2      orange
4    a2  strawberry
5    a2       lemon
6    a3      orange
7    a3      banana
8    a6        meat
9    a7        beer
10   a8       juice

Given a query in input query , I want to match it to query by other user in user query table, and return the top 3 ranked by total number of counts.

For example, orange in input query , it matches user a1 , a2 , a3 in user query where all have queried orange , other items they have query are strawberry (count of 2), pear , lemon , banana (count of 1).

The answer will be strawberry （since it has max count), pear , lemon (since we only return top 3).

Similar reasoning for apple (no user query therefore output 'nothing') and meat query.

So the final output table is

    query   recommend
0  orange  strawberry
1  orange        pear
2  orange       lemon
3   apple     nothing
4    meat     nothing

What's the efficient way to do that given user query have 1 million rows?

here's the code for input query , user query and output table

df_input = pd.DataFrame( {'query': {0: 'orange', 1: 'apple', 2: 'meat'}} )
df_user = pd.DataFrame( {'user': {0: 'a1', 1: 'a1', 2: 'a1', 3: 'a2', 4: 'a2', 5: 'a2', 6: 'a3', 7: 'a3', 8: 'a6', 9: 'a7', 10: 'a8'}, 'query': {0: 'orange', 1: 'strawberry', 2: 'pear', 3: 'orange', 4: 'strawberry', 5: 'lemon', 6: 'orange', 7: 'banana', 8: 'meat', 9: 'beer', 10: 'juice'}} )
df_output = pd.DataFrame( {'query': {0: 'orange', 1: 'orange', 2: 'orange', 3: 'apple', 4: 'meat'}, 'recommend': {0: 'strawberry', 1: 'pear', 2: 'lemon', 3: 'nothing', 4: 'nothing'}} )

Answer 1

Depending on a memory resource you have, choose either of the following solutions.

Code:

# Preparation:

import pandas as pd

# Create sample dataframes
df_input = pd.DataFrame({'query': {0: 'orange', 1: 'apple', 2: 'meat'}})
df_user = pd.DataFrame({'user': {0: 'a1', 1: 'a1', 2: 'a1', 3: 'a2', 4: 'a2', 5: 'a2', 6: 'a3', 7: 'a3', 8: 'a6', 9: 'a7', 10: 'a8'}, 'query': {0: 'orange', 1: 'strawberry', 2: 'pear', 3: 'orange', 4: 'strawberry', 5: 'lemon', 6: 'orange', 7: 'banana', 8: 'meat', 9: 'beer', 10: 'juice'}})

# Define how many recommended items you need for each query
n_top = 3

# Exclude unnecessary rows for caluculation
dfu = df_user.drop_duplicates()
queries = df_input['query']
users = dfu.loc[dfu['query'].isin(queries), 'user'].drop_duplicates()
mask_q = dfu['query'].isin(queries)
mask_u = dfu['user'].isin(users)
df1 = dfu[mask_u&mask_q].set_index('user')
df2 = dfu[mask_u].set_index('user')

# Solution 1:

If you have a large memory resource, try the following code.

# Carry out the basket analysis
df = df1.join(df2, lsuffix='_x', rsuffix='_y')
df = df[df.query_x!=df.query_y].reset_index()
df = df.groupby(['query_x', 'query_y'], as_index=False).count()
df = df.sort_values('user', ascending=False).groupby('query_x').head(n_top)
df = df.drop('user', axis=1).rename(columns={'query_x': 'query', 'query_y': 'recommend'})
df = df_input.merge(df, how='left', on='query').fillna('nothing')

# Solution 2:

If you have a limitation of a memory resource, try the following code. It takes much longer than the solution 1, but you can complete the calculation almost certainly.

# Carry out the basket analysis
df = pd.DataFrame()
for _, df_q1 in df1.groupby('query'):
    _df = pd.DataFrame()
    for _, df_q2 in df2.groupby('query'):
        df_q1q2 = df_q1.join(df_q2, lsuffix='_x', rsuffix='_y')
        df_q1q2 = df_q1q2.reset_index().groupby(['query_x', 'query_y'], as_index=False).count()
        _df = _df.append(df_q1q2)
    _df = _df[_df.query_x!=_df.query_y]
    _df = _df.sort_values('user', ascending=False).groupby('query_x').head(n_top)
    df = df.append(_df)
df = df.drop('user', axis=1).rename(columns={'query_x': 'query', 'query_y': 'recommend'})
df = df_input.merge(df, how='left', on='query').fillna('nothing')

Output of both solution 1 and 2:

	query	recommend
0	orange	strawberry
1	orange	banana
2	orange	lemon
3	apple	nothing
4	meat	nothing