评估列表相似性

Question

I have a dataframe that contains columns of various item recommendations, and the elements are represented as a list (in reality all lists have 10 elements in them, but this is less important):

user_id     actual            predicted            popular             random
u1          [a,b,c]           [a,b,d]              [c,e,f]             [d,e,f]
u2          [a,b,d]           [a,b,c]              [c,e,f]             [a,b,c]
u3          [c,e,f]           [a,c,e]              [c,e,f]             [a,c,f]
u4          [c,e,f]           [a,e,f]              [c,e,f]             [a,d,f]
u5          [b,e,f]           [a,b,e]              [c,e,f]             [a,c,e]  

While I have some separate statistics regarding predicted , I would like to compare how close to the actual lists these predicted , popular and random lists are. popular has always the same three items.

I was thinking of calculating the percentages for each case and then averaging:

 user_id                predicted            popular             random
 u1                     0.66                 0.33                0
 u2                     0.66                 0                   0.33
 u3                     0.66                 1                   0.66
 u4                     0.66                 1                   0.33
 u5                     0.66                 0.33                0.33

Normally, I would do something like:

setA = set(listA)
setB = set(listB)

overlap = setA & setB
universe = setA | setB

result = float(len(overlap)) / len(setA) * 100

But how can I do this for a large dataframe?

Answer 1

So, given the following dataframe:

import pandas as pd

df = pd.DataFrame(
    {
        "user_id": {0: "u1", 1: "u2", 2: "u3", 3: "u4", 4: "u5"},
        "actual": {
            0: ["a", "b", "c"],
            1: ["a", "b", "d"],
            2: ["c", "e", "f"],
            3: ["c", "e", "f"],
            4: ["b", "e", "f"],
        },
        "predicted": {
            0: ["a", "b", "d"],
            1: ["a", "b", "c"],
            2: ["a", "c", "e"],
            3: ["a", "e", "f"],
            4: ["a", "b", "e"],
        },
        "popular": {
            0: ["c", "e", "f"],
            1: ["c", "e", "f"],
            2: ["c", "e", "f"],
            3: ["c", "e", "f"],
            4: ["c", "e", "f"],
        },
        "random": {
            0: ["d", "e", "f"],
            1: ["a", "b", "c"],
            2: ["a", "c", "f"],
            3: ["a", "d", "f"],
            4: ["a", "c", "e"],
        },
    }
)

You could try this:

# Convert lists into sets
df = df.applymap(lambda x: set(x) if isinstance(x, list) else x)

# Iterate to create new columns with percentages
for i in range(df.shape[0]):
    for col in ["predicted", "popular", "random"]:
        df.loc[i, f"{col}_pct"] = (
            len(df.loc[i, "actual"] & df.loc[i, col]) / len(df.loc[i, "actual"]) * 100
        )

# Cleanup
df = df[["user_id", "predicted_pct", "popular_pct", "random_pct"]]

And here is the expected result:

print(df)
# Outputs
  user_id  predicted_pct  popular_pct  random_pct
0      u1      66.666667    33.333333    0.000000
1      u2      66.666667     0.000000   66.666667
2      u3      66.666667   100.000000   66.666667
3      u4      66.666667   100.000000   33.333333
4      u5      66.666667    66.666667   33.333333