[英]Retrieving data from a yaml file based on a Python list
我在 ipython 工作; 我有一个 Yaml 文件和一个与我的 Yaml 文件对应的 [thomas] id 列表(thomas:文件的第三行)。 下面只是文件的一小部分。 完整的文件可以在这里找到( https://github.com/108michael/congress-legislators/blob/master/legislators-historical.yaml )
- id:
bioguide: C000858
thomas: '00246'
lis: S215
govtrack: 300029
opensecrets: N00002091
votesmart: 53288
icpsr: 14809
fec:
- S0ID00057
wikipedia: Larry Craig
house_history: 11530
name:
first: Larry
middle: E.
last: Craig
bio:
birthday: '1945-07-20'
gender: M
religion: Methodist
terms:
- type: rep
start: '1981-01-05'
end: '1983-01-03'
state: ID
district: 1
party: Republican
- type: rep
start: '1983-01-03'
end: '1985-01-03'
state: ID
district: 1
party: Republican
我想解析该文件,并为我的列表中与 [thomas:] 中的 Id 对应的每个 id 我想检索以下内容:[fec] :(可能不止一个,我需要所有这些) [姓名:] [第一个:] [中间:] [最后一个:]; [生物:] [生日:]; [terms:](很可能有多个词条,我需要所有词条)[type:] [start:] [state:] [party:]。 最后,也可能存在 fec 数据不可用的情况。
1)我应该如何存储数据? 我对 Python(我的第一种编程语言)还是比较陌生,不确定如何存储数据。 直觉上,我会说字典; 然而,最重要的是易于访问和数据检索。 以前,我将类似的嵌套数据存储为 csv。 这种方法看起来有点笨重。 如果我可以制作一个字典(我正在检索的数据)的列表(来自我拥有的 thomas id),这似乎是理想的。
2) 我不确定如何设置 for/while 语句,以便我只检索与我的 thomas id 列表相对应的数据。
我开始编写我期望将信息写入 CSV 的代码:
import pandas as pd
import yaml
import glob
import CSV
df = pd.concat((pd.read_csv(f, names=['date','bill_id','sponsor_id']) for f in glob.glob('/home/jayaramdas/anaconda3/df/s11?_s_b')))
outputfile = open('sponsor_details', 'W', newline='')
outputwriter = csv.writer(outputfile)
df = df.drop_duplicates('sponsor_id')
sponsor_list = df['sponsor_id'].tolist()
with open('legislators-historical.yaml', 'r') as f:
data = yaml.load(f)
for sponsor in sponsor_list:
where sponsor == data[0]['thomas']:
x = data[0]['thomas']
a = data[0]['name']['first']
b = data[0]['name']['middle']
c = data[0]['name']['last']
d = data[0]['bio']['gender']
e = data[0]['bio']['religion']
for fec in data[0]['id']:
c = fec.get('fec')
for terms in data[0]['id']:
t = terms.get('type')
s = terms.get('start')
state = terms.get('state')
p = terms.get('party')
outputwriter.writerow([x, a, b, c, d, e, c, t, s, state, p])
outputfile.flush()
我收到以下错误:
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-48-057d25de7e11> in <module>()
15
16 for sponsor in sponsor_list:
---> 17 if sponsor == data[0]['thomas']:
18 x = data[0]['thomas']
19 a = data[0]['name']['first']
KeyError: 'thomas'
我认为您可以尝试解析 YAML 并将其加载到数据框,对其进行规范化:
import pandas as pd
from yaml import safe_load
with open('legislators-historical.yaml', 'r') as f:
df = pd.json_normalize(safe_load(f))
print(df.head())
输出:
bio.birthday bio.gender bio.religion id.bioguide id.fec id.govtrack \
0 1943-12-02 M Protestant A000109 [S6CO00168] 300003
1 1745-04-02 M NaN B000226 NaN 401222
2 1742-03-21 M NaN B000546 NaN 401521
3 1743-06-16 M NaN B001086 NaN 402032
4 1730-07-22 M NaN C000187 NaN 402334
id.house_history id.icpsr id.lis id.opensecrets id.thomas id.votesmart \
0 8410 29108 S250 N00009082 00011 26783
1 NaN 507 NaN NaN NaN NaN
2 9479 786 NaN NaN NaN NaN
3 10177 1260 NaN NaN NaN NaN
4 10687 1538 NaN NaN NaN NaN
id.wikipedia name.first name.last name.middle \
0 Wayne Allard Wayne Allard A.
1 NaN Richard Bassett NaN
2 NaN Theodorick Bland NaN
3 Aedanus Burke Aedanus Burke NaN
4 Daniel Carroll Daniel Carroll NaN
terms
0 [{'party': 'Republican', 'type': 'rep', 'state...
1 [{'party': 'Anti-Administration', 'type': 'sen...
2 [{'end': '1791-03-03', 'district': 9, 'type': ...
3 [{'end': '1791-03-03', 'district': 2, 'type': ...
4 [{'end': '1791-03-03', 'district': 6, 'type': ...
更新:
以下版本将过滤您的输入数据,因此只会处理包含“thomas”和“fec”的记录:
import pandas as pd
from yaml import safe_load
def read_yaml(fn):
with open(fn, 'r') as fi:
return safe_load(fi)
def filter_data(data):
result_data = []
for x in data:
if 'id' not in x: continue
if 'fec' not in x['id']: continue
if 'thomas' not in x['id']: continue
result_data.append(x)
return result_data
fn = 'aaa.yaml'
df = pd.json_normalize(filter_data(read_yaml(fn)), 'terms', [['id', 'fec'], ['id', 'thomas']])
print(df.head())
df.to_csv('out.csv')
输出:
class district end party start state type \
0 NaN 4 1993-01-03 Republican 1991-01-03 CO rep
1 NaN 4 1995-01-03 Republican 1993-01-05 CO rep
2 NaN 4 1997-01-03 Republican 1995-01-04 CO rep
3 2 NaN 2003-01-03 Republican 1997-01-07 CO sen
4 2 NaN 2009-01-03 Republican 2003-01-07 CO sen
url id.thomas id.fec
0 NaN 00011 S6CO00168
1 NaN 00011 S6CO00168
2 NaN 00011 S6CO00168
3 NaN 00011 S6CO00168
4 http://allard.senate.gov 00011 S6CO00168
PS,如您所见,这将复制您的行(请参阅: id.thomas
和id.fec
),以便它可以显示为数据框
更新2
您可能还想将“id.fec”中的列表转换为列,但我会在其他数据框中进行:
df_fec = df['id.fec'].apply(pd.Series)
print(df_fec.head())
输出:
0 1
0 S8AR00112 H2AR01022
1 S8AR00112 H2AR01022
2 S8AR00112 H2AR01022
3 S8AR00112 H2AR01022
4 S6CO00168 NaN
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.