[英]compare list of dicts and merge to one python
I am writing a python program to merge similar dicts to one.我正在编写一个 python 程序来将类似的字典合并为一个。 I have list of two dicts.我有两个字典的列表。 i have a empty filter_cache
dict.我有一个空的filter_cache
字典。 i have input_completed =false
variable.我有input_completed =false
变量。 I will get the dicts using for loop.我将使用 for 循环获取 dicts。 After the first dict gets entered, it gets processed and merged the similar in to one and i assign the result in to the filter_cache
dict.在输入第一个 dict 后,它会被处理并将相似的内容合并为一个,然后我将结果分配给filter_cache
dict。 Next dict from the list entered through the for loop,it gets processed and combined it to one.通过 for 循环输入的列表中的下一个 dict 得到处理并将其合并为一个。 After that, i change input_completed
to true
due to the empty of data
.之后,由于data
为空,我将input_completed
更改为true
。 Next, i want to compare the previous dict in the filter_cache
and the present filter_cache
and then combine it again using some function and assign it to the fiter_cache
.接下来,我想比较filter_cache
的前一个 dict 和当前的filter_cache
,然后使用某个函数将其再次组合并将其分配给fiter_cache
。 How to make it possible如何使它成为可能
Here's the code:这是代码:
from itertools import groupby
field_to_be_check = "state"
merger = ["city", "haps"]
merge_name = ["cities", "my_haps"]
data = [
[{'haps': 'hap0', 'state': 'tamil nadu','ads': 'ad1', 'city': 'tenkasi'},
{'haps': 'hap0', 'state': 'tamil nadu','ads': 'ad4', 'city': 'nagerkoil'},
{'haps': 'hap0', 'state': 'tamil nadu','ads': 'ad1', 'city': 'tuticorin'},
{'haps': 'hap1', 'state': 'kerala', 'ads': 'ad2', 'city': 'kolikodu'},
{'haps': 'hap1', 'state': 'kerala', 'ads': 'ad2', 'city': 'kottayam'},
{'haps': 'hap1', 'state': 'kerala', 'ads': 'ad2', 'city': 'idukki'},
{'haps': 'hap2', 'state': 'mumbai', 'ads': 'ad3', 'city': 'Akola'},
{'haps': 'hap2', 'state': 'mumbai', 'ads': 'ad3', 'city': 'Washim'},
{'haps': 'hap2', 'state': 'mumbai', 'ads': 'ad3', 'city': 'Jalna'},
{'haps': 'hap2', 'state': 'mumbai', 'ads': 'ad3', 'city': 'Latur'}],
[{'haps': 'hap1', 'state': 'tamil nadu','ads': 'ad1', 'city': 'madurai'},
{'haps': 'hap0', 'state': 'tamil nadu','ads': 'ad1', 'city': 'chennai'},
{'haps': 'hap1', 'state': 'kerala', 'ads': 'ad2', 'city': 'palakad'},
{'haps': 'hap1', 'state': 'kerala', 'ads': 'ad2', 'city': 'guruvayor'},
{'haps': 'hap2', 'state': 'mumbai', 'ads': 'ad3', 'city': 'Nanded'},
]
]
input_completed = False
filter_cache = {}
for datas in data:
#Function to make the merger lists
def process_group(group, merger_item):
item_set = set()
item_list = []
for item in group:
item_set.add(item[merger_item])
for item in item_set:
item_list.append({merger_item: item})
return item_list
#Group on haps, state and ads
grp = groupby(datas,key=lambda x:(x[field_to_be_check]))
result = []
#Iterate through the group and build your result list
for model, group in grp:
cities_dict = {}
cities_dict[field_to_be_check] = model
group_list = list(group)
#Make the list for merger fields
for idx, name in enumerate(merger):
cities_dict[merge_name[idx]] = process_group(group_list, name)
result.append(cities_dict)
filter_cache.update({'aggregate':result})
print(filter_cache)
final_result = filter_cache
Output i got:我得到的输出:
{'aggregate': [{'state': 'tamil nadu', 'my_haps': [{'haps': 'hap0'}], 'cities': [{'city': 'tuticorin'}, {'city': 'tenkasi'}, {'city': 'nagerkoil'}]}, {'state': 'kerala', 'my_haps': [{'haps': 'hap1'}], 'cities': [{'city': 'idukki'}, {'city': 'kolikodu'}, {'city': 'kottayam'}]}, {'state': 'mumbai', 'my_haps': [{'haps': 'hap2'}], 'cities': [{'city': 'Akola'}, {'city': 'Jalna'}, {'city': 'Latur'}, {'city': 'Washim'}]}]}
{'aggregate': [{'state': 'tamil nadu', 'my_haps': [{'haps': 'hap1'}, {'haps': 'hap0'}], 'cities': [{'city': 'madurai'}, {'city': 'chennai'}]}, {'state': 'kerala', 'my_haps': [{'haps': 'hap1'}], 'cities': [{'city': 'palakad'}, {'city': 'guruvayor'}]}, {'state': 'mumbai', 'my_haps': [{'haps': 'hap2'}], 'cities': [{'city': 'Nanded'}]}]}
Required Output:所需输出:
{'aggregate': [{'state': 'tamil nadu', 'my_haps': [{'haps': 'hap0'},{'haps': 'hap1'}], 'cities': [{'city': 'tuticorin'}, {'city': 'tenkasi'}, {'city': 'nagerkoil'},{'city': 'madurai'}, {'city': 'chennai'}]}, {'state': 'kerala', 'my_haps': [{'haps': 'hap1'}], 'cities': [{'city': 'idukki'}, {'city': 'kolikodu'}, {'city': 'kottayam'},{'city': 'palakad'}, {'city': 'guruvayor'}]}, {'state': 'mumbai', 'my_haps': [{'haps': 'hap2'}], 'cities': [{'city': 'Akola'}, {'city': 'Jalna'}, {'city': 'Latur'}, {'city': 'Washim'},{'city': 'Nanded'}]}]}
Function:功能:
def group_dicts_from_list(lst, group_by, merge_rules, result):
if not lst or not isinstance(lst, list) or not group_by or not merge_rules or \
not isinstance(merge_rules, dict) or not isinstance(result, dict):
return
if "aggregate" not in result:
result["aggregate"] = []
for item in lst:
if isinstance(item, dict):
if group_by in item:
for res in result["aggregate"]:
if res[group_by] == item[group_by]:
tmp = res
break
else:
tmp = {group_by: item[group_by]}
result["aggregate"].append(tmp)
for src, dst in merge_rules.items():
if src in item:
if dst not in tmp:
tmp[dst] = []
src_tmp = {src: item[src]}
if src_tmp not in tmp[dst]:
tmp[dst].append(src_tmp)
elif isinstance(item, list):
group_dicts_from_list(item, group_by, merge_rules, result) # !!! recursion !!!
Usage:用法:
field_to_be_check = "state"
my_merge_rules = {
"city": "cities",
"haps": "my_haps",
}
data = [
[{'haps': 'hap0', 'state': 'tamil nadu', 'ads': 'ad1', 'city': 'tenkasi'},
{'haps': 'hap0', 'state': 'tamil nadu', 'ads': 'ad4', 'city': 'nagerkoil'},
{'haps': 'hap0', 'state': 'tamil nadu', 'ads': 'ad1', 'city': 'tuticorin'},
{'haps': 'hap1', 'state': 'kerala', 'ads': 'ad2', 'city': 'kolikodu'},
{'haps': 'hap1', 'state': 'kerala', 'ads': 'ad2', 'city': 'kottayam'},
{'haps': 'hap1', 'state': 'kerala', 'ads': 'ad2', 'city': 'idukki'},
{'haps': 'hap2', 'state': 'mumbai', 'ads': 'ad3', 'city': 'Akola'},
{'haps': 'hap2', 'state': 'mumbai', 'ads': 'ad3', 'city': 'Washim'},
{'haps': 'hap2', 'state': 'mumbai', 'ads': 'ad3', 'city': 'Jalna'},
{'haps': 'hap2', 'state': 'mumbai', 'ads': 'ad3', 'city': 'Latur'}],
[{'haps': 'hap1', 'state': 'tamil nadu', 'ads': 'ad1', 'city': 'madurai'},
{'haps': 'hap0', 'state': 'tamil nadu', 'ads': 'ad1', 'city': 'chennai'},
{'haps': 'hap1', 'state': 'kerala', 'ads': 'ad2', 'city': 'palakad'},
{'haps': 'hap1', 'state': 'kerala', 'ads': 'ad2', 'city': 'guruvayor'},
{'haps': 'hap2', 'state': 'mumbai', 'ads': 'ad3', 'city': 'Nanded'},
]
]
result = {}
group_dicts_from_list(data, field_to_be_check, my_merge_rules, result)
print(result)
Output:输出:
{'aggregate': [{'state': 'tamil nadu', 'cities': [{'city': 'tenkasi'}, {'city': 'nagerkoil'}, {'city': 'tuticorin'}, {'city': 'madurai'}, {'city': 'chennai'}], 'my_haps': [{'haps': 'hap0'}, {'haps': 'hap1'}]}, {'state': 'kerala', 'cities': [{'city': 'kolikodu'}, {'city': 'kottayam'}, {'city': 'idukki'}, {'city': 'palakad'}, {'city': 'guruvayor'}], 'my_haps': [{'haps': 'hap1'}]}, {'state': 'mumbai', 'cities': [{'city': 'Akola'}, {'city': 'Washim'}, {'city': 'Jalna'}, {'city': 'Latur'}, {'city': 'Nanded'}], 'my_haps': [{'haps': 'hap2'}]}]}
Comments:注释:
I've replaced我换了
merger = ["city", "haps"]
merge_name = ["cities", "my_haps"]
with和
my_merge_rules = {
"city": "cities",
"haps": "my_haps",
}
cause I found it more reliable (it prevents situations when list lengths differs) .因为我发现它更可靠(它可以防止列表长度不同的情况) 。
Your code and the accepted answer seem overcomplicated to me (maybe I did not understand the requirements).您的代码和接受的答案对我来说似乎过于复杂(也许我不理解要求)。 Why not build a dictionary: field_to_be_check -> merger_name -> merger values
and then convert it to the desired format?为什么不建立一个字典: field_to_be_check -> merger_name -> merger values
,然后将其转换为所需的格式?
Creation of the dictionary:字典的创建:
import itertools
data = [[{'haps': 'hap0', 'state': 'tamil nadu','ads': 'ad1', 'city': 'tenkasi'},
{'haps': 'hap0', 'state': 'tamil nadu','ads': 'ad4', 'city': 'nagerkoil'},
{'haps': 'hap0', 'state': 'tamil nadu','ads': 'ad1', 'city': 'tuticorin'},
{'haps': 'hap1', 'state': 'kerala', 'ads': 'ad2', 'city': 'kolikodu'},
{'haps': 'hap1', 'state': 'kerala', 'ads': 'ad2', 'city': 'kottayam'},
{'haps': 'hap1', 'state': 'kerala', 'ads': 'ad2', 'city': 'idukki'},
{'haps': 'hap2', 'state': 'mumbai', 'ads': 'ad3', 'city': 'Akola'},
{'haps': 'hap2', 'state': 'mumbai', 'ads': 'ad3', 'city': 'Washim'},
{'haps': 'hap2', 'state': 'mumbai', 'ads': 'ad3', 'city': 'Jalna'},
{'haps': 'hap2', 'state': 'mumbai', 'ads': 'ad3', 'city': 'Latur'}],
[{'haps': 'hap1', 'state': 'tamil nadu','ads': 'ad1', 'city': 'madurai'},
{'haps': 'hap0', 'state': 'tamil nadu','ads': 'ad1', 'city': 'chennai'},
{'haps': 'hap1', 'state': 'kerala', 'ads': 'ad2', 'city': 'palakad'},
{'haps': 'hap1', 'state': 'kerala', 'ads': 'ad2', 'city': 'guruvayor'},
{'haps': 'hap2', 'state': 'mumbai', 'ads': 'ad3', 'city': 'Nanded'},]]
field_to_be_check = "state"
name_by_merger = {"city": "cities", "haps": "my_haps"}
d = {}
for row in itertools.chain(*data):
inner_d = d.setdefault(row[field_to_be_check], {})
for m, n in name_by_merger.items():
inner_d.setdefault(n, set()).add(row[m])
Content of d
: d
内容:
{'tamil nadu': {'cities': {'chennai', 'nagerkoil', 'tuticorin', 'madurai', 'tenkasi'}, 'my_haps': {'hap0', 'hap1'}}, 'kerala': {'cities': {'kolikodu', 'palakad', 'idukki', 'guruvayor', 'kottayam'}, 'my_haps': {'hap1'}}, 'mumbai': {'cities': {'Nanded', 'Latur', 'Jalna', 'Akola', 'Washim'}, 'my_haps': {'hap2'}}}
The function itertools.chain
gorups the two lists.函数itertools.chain
合并了两个列表。 The setdefault
method creates a new entry in the dictionary if needed.如果需要, setdefault
方法会在字典中创建一个新条目。
The conversion to the desired format is just a(n ugly) dict comprehension:转换为所需格式只是一个(丑陋的)字典理解:
{'aggregate': [{field_to_be_check: k, **{n: [{m: x} for x in v[n]] for m, n in name_by_merger.items()}} for k, v in d.items()]}
Output:输出:
{'aggregate': [{'state': 'tamil nadu', 'cities': [{'city': 'chennai'}, {'city': 'nagerkoil'}, {'city': 'tuticorin'}, {'city': 'madurai'}, {'city': 'tenkasi'}], 'my_haps': [{'haps': 'hap0'}, {'haps': 'hap1'}]}, {'state': 'kerala', 'cities': [{'city': 'kolikodu'}, {'city': 'palakad'}, {'city': 'idukki'}, {'city': 'guruvayor'}, {'city': 'kottayam'}], 'my_haps': [{'haps': 'hap1'}]}, {'state': 'mumbai', 'cities': [{'city': 'Nanded'}, {'city': 'Latur'}, {'city': 'Jalna'}, {'city': 'Akola'}, {'city': 'Washim'}], 'my_haps': [{'haps': 'hap2'}]}]}
Of course you can build the desired output in one pass, but since the format is cumbersome, I think it's cleaner to create first a nive dictionary and then comply to this format.当然,您可以一次性构建所需的输出,但是由于格式繁琐,我认为首先创建一个 nive 字典然后遵守这种格式会更清晰。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.