[英]Python iterate through list of dictionaries
I have the below list of dictionaries -我有以下字典列表 -
results = [
{'type': 'check_datatype',
'kwargs': {'table': 'cars', 'columns': ['car_id','index'], 'd_type': 'str'},
'datasource_path': '/cars_dataset_ok/',
'Result': False},
{'type': 'check_string_consistency',
'kwargs': {'table': 'cars', 'columns': ['car_id'], 'string_length': 6},
'datasource_path': '/cars_dataset_ok/',
'Result': False}
]
I want output list with below output where key and value fields are coming from kwargs key in the above list -我想要 output 列表,下面是 output ,其中键和值字段来自上面列表中的 kwargs 键-
id|key|value|index id|键|值|索引
[[1,table,cars,null],[1,columns,car_id,1],[1,columns,index,2] [1,dtype,str,null],[2,table,cars,null],[2,columns,car_id,null],[2,string_length,6,null]] [[1,table,cars,null],[1,columns,car_id,1],[1,columns,index,2] [1,dtype,str,null],[2,table,cars,null], [2,columns,car_id,null],[2,string_length,6,null]]
Update - Now, i want one more column in output - uniquehaschode --> here unique hashcode means Dictionaries with the same keys and values should generate the same id or hash.更新 - 现在,我想要 output 中的另一列 - uniquehaschode -> 这里唯一的哈希码意味着具有相同键和值的字典应该生成相同的 id 或 hash。 Hence if key value pairs are same in dictionary 'kwargs', then they should return the same hashcode.因此,如果字典“kwargs”中的键值对相同,那么它们应该返回相同的哈希码。 Output should be like this - Output 应该是这样的 -
[[1,table,cars,null,uniquehaschode1],[1,columns,car_id,1,uniquehaschode1],[1,columns,index,2,uniquehaschode1] [1,dtype,str,null,uniquehaschode1],[2,table,cars,null,uniquehaschode2],[2,columns,car_id,null,uniquehaschode2],[2,string_length,6,null,uniquehaschode2]] [[1,table,cars,null,uniquehaschode1],[1,columns,car_id,1,uniquehaschode1],[1,columns,index,2,uniquehaschode1] [1,dtype,str,null,uniquehaschode1,[uniquehaschode1] ,表,汽车,null,uniquehaschode2],[2,columns,car_id,null,uniquehaschode2],[2,string_length,6,Z37A6259CC0C1DAE299A78ha]66
Also, i don't want to insert anything into this table if a particular uniquehaschode already exists.另外,如果已经存在特定的 uniquehaschode,我不想在此表中插入任何内容。
Update2: I want to create a dataframe with below schema. Update2:我想创建一个具有以下架构的 dataframe。 args_id will be same for each unique pair of (kwargs and check_name). args_id 对于每对唯一的(kwargs 和 check_name)都是相同的。 i want to run the above list of dictionaries everyday and hence for different date run, args_id should be same if unique pair of (kwargs and check_name) has come again.我想每天运行上面的字典列表,因此对于不同的日期运行,如果唯一的(kwargs 和 check_name)对再次出现,args_id 应该相同。 i want to store this result into a dataframe everyday and then put it into my delta table of spark.我想每天将此结果存储到 dataframe 中,然后将其放入我的 spark 增量表中。
Type|time|args_id
check_datatype|2021-03-29|0
check_string_consistency|2021-03-29|1
check_datatype|2021-03-30|0
Until now, i was using below code -直到现在,我一直在使用下面的代码 -
type_results = [[elt['type'] for
elt in results]
checkColumns = ['type']
spark = SparkSession.builder.getOrCreate()
DF = spark.createDataFrame(data=results, schema=checkColumns)
DF = DF.withColumn("time", F.current_timestamp())
DF = DF.withColumn("args_id", F.row_number().over(Window.orderBy(F.monotonically_increasing_id())))
results = [
{'type': 'check_datatype',
'kwargs': {'table': 'cars', 'columns': ['car_id','index'], 'd_type': 'str'},
'datasource_path': '/cars_dataset_ok/',
'Result': False},
{'type': 'check_string_consistency',
'kwargs': {'table': 'cars', 'columns': ['car_id'], 'string_length': 6},
'datasource_path': '/cars_dataset_ok/',
'Result': False}
]
for each in results:
print(each['kwargs'])
Probably you need:可能你需要:
results = [
{'type': 'check_datatype',
'kwargs': {'table': 'cars', 'columns': ['car_id','index'], 'd_type': 'str'},
'datasource_path': '/cars_dataset_ok/',
'Result': False},
{'type': 'check_string_consistency',
'kwargs': {'table': 'cars', 'columns': ['car_id'], 'string_length': 6},
'datasource_path': '/cars_dataset_ok/',
'Result': False}
]
result_list = []
for c, l in enumerate(results, start=1):
for key, value in l['kwargs'].items():
if isinstance(value,list):
if len(value) == 1:
result_list.append([str(c),key,value[0],'null'])
continue
for i in value:
result_list.append([str(c),key,i,str(value.index(i)+1)])
else:
result_list.append([str(c),key,value,'null'])
print(result_list)
Output: Output:
[['1', 'table', 'cars', 'null'], ['1', 'columns', 'car_id', '1'], ['1', 'columns', 'index', '2'], ['1', 'd_type', 'str', 'null'], ['2', 'table', 'cars', 'null'], ['2', 'columns', 'car_id', 'null'], ['2', 'string_length', 6, 'null']]
As for the Update part you can use pip install maps
:至于更新部分,您可以使用pip install maps
:
import maps
results = [
{'type': 'check_datatype',
'kwargs': {'table': 'cars', 'columns': ['car_id','index'], 'd_type': 'str'},
'datasource_path': '/cars_dataset_ok/',
'Result': False},
{'type': 'check_string_consistency',
'kwargs': {'table': 'cars', 'columns': ['car_id'], 'string_length': 6},
'datasource_path': '/cars_dataset_ok/',
'Result': False},
{'type': 'check_string_consistency',
'kwargs': {'table': 'cars', 'columns': ['car_id'], 'string_length': 6},
'datasource_path': '/cars_dataset_ok/',
'Result': False}
]
result_list = []
for c, l in enumerate(results, start=1):
h = hash(maps.FrozenMap.recurse(l['kwargs']))
for key, value in l['kwargs'].items():
if isinstance(value,list):
if len(value) == 1:
result_list.append([str(c),key,value[0],'null', f'{h}-{c}'])
continue
for i in value:
result_list.append([str(c),key,i,str(value.index(i)+1),f'{h}-{c}'])
else:
result_list.append([str(c),key,value,'null',f'{h}-{c}'])
print(result_list)
Output: Output:
[['1', 'table', 'cars', 'null', '-6654319495930648246-1'], ['1', 'columns', 'car_id', '1', '-6654319495930648246-1'], ['1', 'columns', 'index', '2', '-6654319495930648246-1'], ['1', 'd_type', 'str', 'null', '-6654319495930648246-1'], ['2', 'table', 'cars', 'null', '-3876605863049152209-2'], ['2', 'columns', 'car_id', 'null', '-3876605863049152209-2'], ['2', 'string_length', 6, 'null', '-3876605863049152209-2'], ['3', 'table', 'cars', 'null', '-3876605863049152209-3'], ['3', 'columns', 'car_id', 'null', '-3876605863049152209-3'], ['3', 'string_length', 6, 'null', '-3876605863049152209-3']]
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.