计算元组列表中的出现次数

Question

I think it is best to start with input and output:

list_of_items = [
    {"A": "abc", "B": "dre", "C": "ccp"},
    {"A": "qwe", "B": "dre", "C": "ccp"},
    {"A": "abc", "B": "dre", "C": "ccp"},
]

result = {'A-abc-->B': {'dre': 2},
          'A-abc-->C': {'ccp': 2},
          'A-qwe-->B': {'dre': 1},
          'A-qwe-->C': {'ccp': 1},
          'B-dre-->A': {'abc': 2, 'qwe': 1},
          'B-dre-->C': {'ccp': 3},
          'C-ccp-->A': {'abc': 2, 'qwe': 1},
          'C-ccp-->B': {'dre': 3}}

My initial input is items that come as a stream. Those items are basically dictionaries with key and value. My goal is to get for each specific key and value the maximum value for all other keys that came with it.

So if out of 100 items, for the key "A" with value "1" I got in 90 items for key "B" the value "2" and in 10 items for key "B" the value "1111" I want to see a list that will show me those numbers. B2=90, B1111=10.

My code is working. But, my real life scenario contains more than 100000 different values for about 20 keys. also, my final goal would be to run this as a job on Flink.

So I am looking for help with Counter / python stream api.

all_tuple_list_items = []
for dict_item in list_of_items:
    list_of_tuples = [(k, v) for (k, v) in dict_item.items()]
    all_tuple_list_items.append(list_of_tuples)

result_dict = {}
for list_of_tuples in all_tuple_list_items:
    for id_tuple in list_of_tuples:
        all_other_tuples = list_of_tuples.copy()
        all_other_tuples.remove(id_tuple)
        dict_of_specific_corresponding = {}

        for corresponding_other_tu in all_other_tuples:
            ids_connection_id = id_tuple[0] + "-" + str(id_tuple[1]) + "-->" + corresponding_other_tu[0]
            corresponding_id = str(corresponding_other_tu[1])

            if result_dict.get(ids_connection_id) is None:
                result_dict[ids_connection_id] = {corresponding_id: 1}
            else:
                if result_dict[ids_connection_id].get(corresponding_id) is None:
                    result_dict[ids_connection_id][corresponding_id] = 1
                else:
                    result_dict[ids_connection_id][corresponding_id] = result_dict[ids_connection_id][
                                                                           corresponding_id] + 1

pprint(result_dict)

Answer 1

You can use the function permutations() to generate all permutations of items in dicts and Counter to count them. Finally you can use defaultdict() to group items from Counter :

from collections import Counter, defaultdict
from itertools import permutations
from pprint import pprint

list_of_items = [
    [{"A": "abc", "B": "dre", "C": "ccp"}],
    [{"A": "qwe", "B": "dre", "C": "ccp"}],
    [{"A": "abc", "B": "dre", "C": "ccp"}],
]

c = Counter(p for i in list_of_items       
              for p in permutations(i[0].items(), 2))
d = defaultdict(dict)
for ((i, j), (k, l)), num in c.items():
    d[f'{i}-{j}-->{k}'][l] = num

pprint(d)

Output:

defaultdict(<class 'dict'>,
            {'A-abc-->B': {'dre': 2},
             'A-abc-->C': {'ccp': 2},
             'A-qwe-->B': {'dre': 1},
             'A-qwe-->C': {'ccp': 1},
             'B-dre-->A': {'abc': 2, 'qwe': 1},
             'B-dre-->C': {'ccp': 3},
             'C-ccp-->A': {'abc': 2, 'qwe': 1},
             'C-ccp-->B': {'dre': 3}})

Answer 2

Got it to work. But, still wants to get a more efficient way. Using counters and stream. is that possible?

code

all_tuple_list_items = []
for dict_item in list_of_items:
    list_of_tuples = [(k, v) for (k, v) in dict_item[0].items()]
    all_tuple_list_items.append(list_of_tuples)

result_dict = {}
for list_of_tuples in all_tuple_list_items:
    for id_tuple in list_of_tuples:
        all_other_tuples = list_of_tuples.copy()
        all_other_tuples.remove(id_tuple)
        dict_of_specific_corresponding = {}

        for corresponding_other_tu in all_other_tuples:
            ids_connection_id = id_tuple[0] + "-" + str(id_tuple[1]) + "-->" + corresponding_other_tu[0]
            corresponding_id = str(corresponding_other_tu[1])

            if result_dict.get(ids_connection_id) is None:
                result_dict[ids_connection_id] = {corresponding_id: 1}
            else:
                if result_dict[ids_connection_id].get(corresponding_id) is None:
                    result_dict[ids_connection_id][corresponding_id] = 1
                else:
                    result_dict[ids_connection_id][corresponding_id] = result_dict[ids_connection_id][
                                                                           corresponding_id] + 1

pprint(result_dict)