有没有更快的方法来查找列表列表中两个元素的共现

Question

i have a list like this.

a = [
    ['a1', 'b2', 'c3'],
    ['c3', 'd4', 'a1'],
    ['b2', 'a1', 'e5'],
    ['d4', 'a1', 'b2'],
    ['c3', 'b2', 'a1']
    ]

I'll be given x (eg: 'a1'). I have to find the co-occurrence of a1 with every other element and sort it and retrieve the top n (eg: top 2) my answer should be

[
 {'product_id': 'b2', 'count': 4}, 
 {'product_id': 'c3', 'count': 3}, 
]

my current code looks like this:

def compute (x):
    set_a = list(set(list(itertools.chain(*a))))
    count_dict = []
    for i in range(0, len(set_a)):
        count = 0
        for j in range(0, len(a)):
            if x==set_a[i]:
                continue
            if x and set_a[i] in a[j]:
                count+=1
        if count>0:
            count_dict.append({'product_id': set_a[i], 'count': count})
    count_dict = sorted(count_dict, key=lambda k: k['count'], reverse=True) [:2]
    return count_dict

And it works beautifully for smaller inputs. However my actual input has 70000 unique items instead of 5 (a to e) and 1.3 million rows instead of 5. And hence mxn becomes very exhaustive. Is there a faster way to do this?

Answer 1

"Faster" is a very general term. Do you need a shorter total processing time, or shorter response time for a request? Is this for only one request, or do you want a system that handles repeated inputs?

If what you need is the fastest response time for repeated inputs, then convert this entire list of lists into a graph, with each element as a node, and the edge weight being the number of occurrences between the two elements. You make a single pass over the data to build the graph. For each node, sort the edge list by weight. From there, each request is a simple lookup: return the weight of the node's top edge, which is a hash (linear function) and two direct access operations (base address + offset).

---

UPDATE after OP's response

"fastest response" seals the algorithm, then. What you want to have is a simple dict, keyed by each node. The value of each node is a sorted list of related elements and their counts.

A graph package (say, networkx ) will give you a good entry to this, but may not retain a node's edges in fast form, nor sorted by weight. Instead, pre-process your data base. For each row, you have a list of related elements. Let's just look at the processing for some row in the midst of the data set; call the elements a5, b2, z1 , and the dict d . Assume that a5, b2 is already in your dict.

using `intertools`, Iterate through the six pairs.
(a5, b2):
    d[a5][b2] += 1
(a5, z1):
    d[a5][z1]  = 1  (creates a new entry under a5)
(b2, a5):
    d[b2][a5] += 1
(b2, z1):
    d[b2][z1]  = 1  (creates a new entry under b2)
(z1, a5):
    d[z1] = {}      (creates a new z1 entry in d)
    d[z1][a5]  = 1  (creates a new entry under z1)
(z1, b2):
    d[z1][b2]  = 1  (creates a new entry under z1)

You'll want to use defaultdict to save you some hassle to detect and initialize new entries.

With all of that handled, you now want to sort each of those sub-dicts into order based on the sub-level values. This leaves you with an ordered sequence for each element. When you need to access the top n connected elements, you go straight to the dict and extract them:

top = d[elem][:n]

Can you finish the coding from there?

Answer 2

as mentioned by @prune it is not mentioned that do you want a shorter processing time or shorter response time. So I will explain two approach to this problem

1. The optimised code approach (for less processing time)

from heapq import nlargest
from operator import itemgetter

#say we have K THREADS
def compute (x,    top_n=2):
    # first you find the unique items and save them somewhere easily accessible
    set_a = list(set(list(itertools.chain(*a))))


    #first find that in which of your ROWS the x exists
    selected_rows=[]
    for i,row in enumerate(a): #this whole loop can be parallelized
        if x in row: 
            selected_rows.append(i) #append index of the row in selected_rows array
    
    
    # time complexity till now is still O(M*N) but this part can be run in parallel as well, as each row       # can be evaluated independently M items can be evaluated independently 
    # THE M rows can be run in parallel, if we have K threads
    # it is going to take us (M/K)*N time complexity to run it.


    count_dict=[]
    
    
    # now the same thing you did earlier but now in second loop we are looking for less rows
    for val in set_a:
        if val==x:
            continue
        count=0
        for ri in selected_rows: # this whole part can be parallelized as well
            if val in a[ri]:
                count+=1
        count_dict.append({'product_id':val, 'count': count})

    # if our selected rows size on worst case is M itself
    # and our unique values are U, the complexity
    # will be (U/K)*(M/K)*N


    

    res = nlargest(top_n, count_dict, key = itemgetter('count'))
    return res

Lets calculate time complexity here If we have K threads then

O((M/K)*N)+O((U/K)*(M/K)*N))

where

M---> Total rows
N---> Total Columns
U---> Unique Values
K---> number of threads

2. Graph approach as suggested by Prune

# other approach
#adding to Prune approach
big_dictionary={}
set_a = list(set(list(itertools.chain(*a))))
for x in set_a:
    big_dictionary[x]=[]
    for y in set_a:
        count=0

        if x==y:
            continue
        for arr in a:
            if (x in arr) and (y in arr):
                count+=1
        big_dictionary[x].append((y,count))

for x in big_dictionary:
    big_dictionary[x]=sorted(big_dictionary[x], key=lambda v:v[1], reverse=True)

Lets calculate time complexity for this one here One time complexity will be:

O(U*U*M*N)

where

M---> Total rows
N---> Total Columns
U---> Unique Values

But once this big_dictionary is calculated once, It will take you just 1 step to get your topN values For example if we want to get top3 values for a1

result=big_dictionary['a1'][:3]

Answer 3

I followed the defaultdict approach as suggested by @Prune above. Here's the final code:

from collections import defaultdict

def recommender(input_item, b_list, n):
    count =[]
    top_items = []
    for x in b.keys():
        lst_2 = b[x]
        common_transactions = len(set(b_list) & set(lst_2))
        count.append(common_transactions)
    
    top_ids = list((np.argsort(count)[:-n-2:-1])[1::])
    top_values_counts = [count[i] for i in top_ids]
    
    key_list = list(b.keys())
    for i,v in enumerate(top_ids):
        item_id = key_list[v]
        top_items.append({item_id:  top_values_counts[i]})
    print(top_items)
    return top_items

a = [
        ['a1', 'b2', 'c3'],
        ['c3', 'd4', 'a1'],
        ['b2', 'a1', 'e5'],
        ['d4', 'a1', 'b2'],
        ['c3', 'b2', 'a1']
    ]

b = defaultdict(list) 
for i, s in enumerate(a):
    for key in s : 
        b[key].append(i)
        
input_item = str(input("Enter the item_id: "))
n = int(input("How many values to be retrieved? (eg: top 5, top 2, etc.): "))
top_items = recommender(input_item, b[input_item], n)

Here's the output for top 3 for 'a1':

[{'b2': 4}, {'c3': 3}, {'d4': 2}]

Thanks!!!