[英]Retrieving data from a yaml file based on a Python list
我在 ipython 工作; 我有一個 Yaml 文件和一個與我的 Yaml 文件對應的 [thomas] id 列表(thomas:文件的第三行)。 下面只是文件的一小部分。 完整的文件可以在這里找到( https://github.com/108michael/congress-legislators/blob/master/legislators-historical.yaml )
- id:
bioguide: C000858
thomas: '00246'
lis: S215
govtrack: 300029
opensecrets: N00002091
votesmart: 53288
icpsr: 14809
fec:
- S0ID00057
wikipedia: Larry Craig
house_history: 11530
name:
first: Larry
middle: E.
last: Craig
bio:
birthday: '1945-07-20'
gender: M
religion: Methodist
terms:
- type: rep
start: '1981-01-05'
end: '1983-01-03'
state: ID
district: 1
party: Republican
- type: rep
start: '1983-01-03'
end: '1985-01-03'
state: ID
district: 1
party: Republican
我想解析該文件,並為我的列表中與 [thomas:] 中的 Id 對應的每個 id 我想檢索以下內容:[fec] :(可能不止一個,我需要所有這些) [姓名:] [第一個:] [中間:] [最后一個:]; [生物:] [生日:]; [terms:](很可能有多個詞條,我需要所有詞條)[type:] [start:] [state:] [party:]。 最后,也可能存在 fec 數據不可用的情況。
1)我應該如何存儲數據? 我對 Python(我的第一種編程語言)還是比較陌生,不確定如何存儲數據。 直覺上,我會說字典; 然而,最重要的是易於訪問和數據檢索。 以前,我將類似的嵌套數據存儲為 csv。 這種方法看起來有點笨重。 如果我可以制作一個字典(我正在檢索的數據)的列表(來自我擁有的 thomas id),這似乎是理想的。
2) 我不確定如何設置 for/while 語句,以便我只檢索與我的 thomas id 列表相對應的數據。
我開始編寫我期望將信息寫入 CSV 的代碼:
import pandas as pd
import yaml
import glob
import CSV
df = pd.concat((pd.read_csv(f, names=['date','bill_id','sponsor_id']) for f in glob.glob('/home/jayaramdas/anaconda3/df/s11?_s_b')))
outputfile = open('sponsor_details', 'W', newline='')
outputwriter = csv.writer(outputfile)
df = df.drop_duplicates('sponsor_id')
sponsor_list = df['sponsor_id'].tolist()
with open('legislators-historical.yaml', 'r') as f:
data = yaml.load(f)
for sponsor in sponsor_list:
where sponsor == data[0]['thomas']:
x = data[0]['thomas']
a = data[0]['name']['first']
b = data[0]['name']['middle']
c = data[0]['name']['last']
d = data[0]['bio']['gender']
e = data[0]['bio']['religion']
for fec in data[0]['id']:
c = fec.get('fec')
for terms in data[0]['id']:
t = terms.get('type')
s = terms.get('start')
state = terms.get('state')
p = terms.get('party')
outputwriter.writerow([x, a, b, c, d, e, c, t, s, state, p])
outputfile.flush()
我收到以下錯誤:
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-48-057d25de7e11> in <module>()
15
16 for sponsor in sponsor_list:
---> 17 if sponsor == data[0]['thomas']:
18 x = data[0]['thomas']
19 a = data[0]['name']['first']
KeyError: 'thomas'
我認為您可以嘗試解析 YAML 並將其加載到數據框,對其進行規范化:
import pandas as pd
from yaml import safe_load
with open('legislators-historical.yaml', 'r') as f:
df = pd.json_normalize(safe_load(f))
print(df.head())
輸出:
bio.birthday bio.gender bio.religion id.bioguide id.fec id.govtrack \
0 1943-12-02 M Protestant A000109 [S6CO00168] 300003
1 1745-04-02 M NaN B000226 NaN 401222
2 1742-03-21 M NaN B000546 NaN 401521
3 1743-06-16 M NaN B001086 NaN 402032
4 1730-07-22 M NaN C000187 NaN 402334
id.house_history id.icpsr id.lis id.opensecrets id.thomas id.votesmart \
0 8410 29108 S250 N00009082 00011 26783
1 NaN 507 NaN NaN NaN NaN
2 9479 786 NaN NaN NaN NaN
3 10177 1260 NaN NaN NaN NaN
4 10687 1538 NaN NaN NaN NaN
id.wikipedia name.first name.last name.middle \
0 Wayne Allard Wayne Allard A.
1 NaN Richard Bassett NaN
2 NaN Theodorick Bland NaN
3 Aedanus Burke Aedanus Burke NaN
4 Daniel Carroll Daniel Carroll NaN
terms
0 [{'party': 'Republican', 'type': 'rep', 'state...
1 [{'party': 'Anti-Administration', 'type': 'sen...
2 [{'end': '1791-03-03', 'district': 9, 'type': ...
3 [{'end': '1791-03-03', 'district': 2, 'type': ...
4 [{'end': '1791-03-03', 'district': 6, 'type': ...
更新:
以下版本將過濾您的輸入數據,因此只會處理包含“thomas”和“fec”的記錄:
import pandas as pd
from yaml import safe_load
def read_yaml(fn):
with open(fn, 'r') as fi:
return safe_load(fi)
def filter_data(data):
result_data = []
for x in data:
if 'id' not in x: continue
if 'fec' not in x['id']: continue
if 'thomas' not in x['id']: continue
result_data.append(x)
return result_data
fn = 'aaa.yaml'
df = pd.json_normalize(filter_data(read_yaml(fn)), 'terms', [['id', 'fec'], ['id', 'thomas']])
print(df.head())
df.to_csv('out.csv')
輸出:
class district end party start state type \
0 NaN 4 1993-01-03 Republican 1991-01-03 CO rep
1 NaN 4 1995-01-03 Republican 1993-01-05 CO rep
2 NaN 4 1997-01-03 Republican 1995-01-04 CO rep
3 2 NaN 2003-01-03 Republican 1997-01-07 CO sen
4 2 NaN 2009-01-03 Republican 2003-01-07 CO sen
url id.thomas id.fec
0 NaN 00011 S6CO00168
1 NaN 00011 S6CO00168
2 NaN 00011 S6CO00168
3 NaN 00011 S6CO00168
4 http://allard.senate.gov 00011 S6CO00168
PS,如您所見,這將復制您的行(請參閱: id.thomas
和id.fec
),以便它可以顯示為數據框
更新2
您可能還想將“id.fec”中的列表轉換為列,但我會在其他數據框中進行:
df_fec = df['id.fec'].apply(pd.Series)
print(df_fec.head())
輸出:
0 1
0 S8AR00112 H2AR01022
1 S8AR00112 H2AR01022
2 S8AR00112 H2AR01022
3 S8AR00112 H2AR01022
4 S6CO00168 NaN
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.