使用ElementTree和BeautifulSoup解析文件：是否可以通過標記級別數來解析文件？

Question

我有這個 xml文件，我基本上想將所有信息記錄到字典中。

我寫了這段代碼：

import requests
import xml.etree.ElementTree as ET
import urllib2
import glob
import pprint
from bs4 import BeautifulSoup

#get the XML file
#response = requests.get('https://www.drugbank.ca/drugs/DB01048.xml')
#with open('output.txt', 'w') as input:
#          input.write(response.content)


#set up lists etc
set_of_files = glob.glob('output*txt')
val = lambda x: "{http://www.drugbank.ca}" + str(x)
key_list = ['drugbank-id','name','description','cas-number','unii','average-mass','monoisotopic-mass','state','indication','pharmacodynamics','mechanism-of-action','toxicity','metabolism','absorption','half-life','protein-binding','route-of-elimination','volume-of-distribution','fda-label','msds']
key_dict = {}
method1 = ['groups','synonyms','patent']
method3_inputs = [('dosages','dosage'),('salts','salt'),('products','product'),('mixtures','mixture'),('packagers','packager'),('categories','category'),('atc-codes','atc-code'),('pdb-entries','pdb-entry'),('food-interactions','food-interaction'),('drug-interactions','drug-interaction'),('properties','property'),('external-identifiers','external-identifier'),('external-links','external-link'),('reactions','reaction')]
list_to_run_thru = ['description','direct-parent','kingdom','superclass','class','subclass']
alternative_parents = []
http_add = '{http://www.drugbank.ca}'
substituents = []
ap_sub = lambda x:'{http://www.drugbank.ca}'+ x

def method2(list2_name,list3_name,list4_name):
     temp_list = []
     for i in subnode:
          if i.tag == list2_name:
               for a in i:
                    if a.tag == list3_name:
                         for u in a:
                              if u.tag == list4_name:
                                   temp_list.append(u.text)
     return temp_list

def method3(list1_name):
     list_of_tuples = []
     for i in subnode:
          if i.tag == list1_name:
               temp_list = []
               for a in i:
                    temp_list.append(a.text)
                    list_of_tuples.append(temp_list)
     return list_of_tuples

new_method3 = []
for i in method3_inputs:
     new_k = http_add + i[0]
     new_v = http_add + i[1]
     new_method3.append((new_k,new_v))

for each_file in set_of_files:
     tree = ET.parse(each_file)
     root = tree.getroot()
     for i in key_list:
          for child in root.getchildren():
               if i not in key_dict:
                    key_dict[i] = [child.find(val(i)).text.encode('utf-8')]
               else:
                    key_dict[i].append(child.find(val(i)).text.encode('utf-8'))

     for node in root:
          for subnode in node:
               if subnode.tag == '{http://www.drugbank.ca}general-references':
                    if 'pubmed-id' not in key_dict:
                         key_dict['pubmed-id'] = method2('{http://www.drugbank.ca}articles','{http://www.drugbank.ca}article','{http://www.drugbank.ca}pubmed-id')
                    else:
                         key_dict['pubmed-id'].append(method2('{http://www.drugbank.ca}articles','{http://www.drugbank.ca}article','{http://www.drugbank.ca}pubmed-id'))

               for a,b in zip(method3_inputs,new_method3):
                    if subnode.tag == b[0]:
                         if a[0] not in key_dict:
                              key_dict[a[0]] = method3(b[1])
                         else:
                              key_dict[method3_inputs].append(method3(b[1]))

               if subnode.tag == '{http://www.drugbank.ca}classification': 
                    for each_item in list_to_run_thru:
                         for i in subnode:
                               if i.tag == ap_sub(each_item):
                                   if i.tag == '{http://www.drugbank.ca}alternative-parent':
                                         alternative_parents.append(i.text)  
                                   if i.tag == '{http://www.drugbank.ca}substituent':
                                         substituents.append(i.text)  

               for i in method1:
                    if subnode.tag == '{http://www.drugbank.ca}' + i:
                         for n in subnode:
                              if i not in key_dict:
                                   key_dict[i] = [n.text]
                              elif i in key_dict:
                                   key_dict[i].append(n.text)

               if subnode.tag == '{http://www.drugbank.ca}pathways':
                    if 'pathways'not in key_dict:
                         key_dict['pathways'] = method2('{http://www.drugbank.ca}pathways','{http://www.drugbank.ca}pathway','{http://www.drugbank.ca}drug')
                    else:
                         key_dict['pathways'].append(method2('{http://www.drugbank.ca}pathways','{http://www.drugbank.ca}pathway','{http://www.drugbank.ca}drug'))


key_dict['alternative_parents'] = alternative_parents
key_dict['substituent'] = substituents


html = requests.get('https://www.drugbank.ca/drugs/DB01048').text
soup = BeautifulSoup(html, 'html.parser')
div_targets = soup.find('div', class_='bond-list-container targets')
targets = div_targets.find_all('div', class_='bond card')

for target in targets:
    k = []
    v = []
    for property in target.find_all('dt'):
        k.append(property.get_text())
    for property in target.find_all('dd'):
        v.append(property.get_text())
    key_dict[target.find('strong').get_text()] = dict(zip(k, v))

print key_dict.keys()

代碼在描述得很好的XML文件上運行。 但是有一些問題我想看看是否有人可以改進：

1. 您可以清楚地看到我不得不對其進行大量硬編碼。 例如，某些標簽的深度為一層（例如，drugbank-id），有些標簽的深度為兩層（例如，組->組），有些標簽的深度為三層（例如，產品，產品，名稱）等。

可以看到，我已經為兩層和三層編寫了一個函數（函數分別稱為method3和method2），然后將其硬編碼在屬於每個函數的列表中。

我想知道是否有人可以向我展示比這更好/更干凈/更有效的代碼。 我的想法是，我有一個ID列表（例如，描述，產品），並且該函數以某種方式理解“這是一個三層標簽（例如，products-> product-> name），因此我不必費勁代碼在所有3層標簽，所有2層標簽等中進行編碼。類似if語句'if標簽為3層深...執行此功能.... if標簽為2層深.. ..做這個'。

2. 免責聲明：我知道此方法同時使用BeautifulSoup和ET進行解析，因為我被困在一節中並在此處獲得了幫助。 不幸的是，我需要精通一個，然后才能繼續，而且我發現HTML版本比XML更加混亂，因此這就是腳本從XML和HTML角度解析文件的原因。

3. 是否有人對如何使腳本更簡潔有任何一般性評論？ 這個想法是將每個“頂級”標簽（即緊接在“毒品類型”標簽下方的所有內容）都放入標簽，然后將所有這些信息基本上下拉到詞典/列表/任何內容中，然后在我想搜索的任何時候某些情況下，我可以在詞典中搜索“最高級”字詞，並且它具有該標簽的子數據，我可以閱讀該標簽（這些級別的組織方式正確，例如，如果標簽下方只有一個級別，則只有一個string是可以的，但是如果標簽下有很多標簽，則可能返回字典/列表/元組/更合適的東西）。

編輯：基於以下幫助，我下載了drugbank.xsd並運行了此命令：

pyxbgen -m DB01048 -u drugbank.xsd --no-validate-changes

然后這個腳本：

from __future__ import print_function
import DB01048

xml = open('DB01048.xml').read()
d_01048 = DB01048.CreateFromDocument(xml)

#print(d_01048.drug[0].state)
#print(d_01048.drug[0].name)
#print(d_01048.drug[0].general_references.articles.article[0].pubmed_id)

和錯誤：

Traceback (most recent call last):
  File "parse_drug_db.py", line 5, in <module>
    d_01048 = DB01048.CreateFromDocument(xml)
  File "/home/drugs/DB01048.py", line 65, in CreateFromDocument
    saxer.parse(io.BytesIO(xmld))
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 110, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/usr/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 213, in feed
    self._parser.Parse(data, isFinal)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 365, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/home/aoidoh/bin/local/lib/python2.7/site-packages/pyxb/binding/saxer.py", line 388, in endElementNS
    binding_object = this_state.endBindingElement()
  File "/home/aoidoh/bin/local/lib/python2.7/site-packages/pyxb/binding/saxer.py", line 245, in endBindingElement
    return self.__bindingInstance._postDOMValidate()
  File "/home/aoidoh/bin/local/lib/python2.7/site-packages/pyxb/binding/basis.py", line 2652, in _postDOMValidate
    self._validateAttributes()
  File "/home/aoidoh/bin/local/lib/python2.7/site-packages/pyxb/binding/basis.py", line 2246, in _validateAttributes
    au.validate(self)
  File "/home/aoidoh/bin/local/lib/python2.7/site-packages/pyxb/binding/content.py", line 251, in validate
    raise pyxb.MissingAttributeError(type(ctd_instance), self.__name, ctd_instance)
pyxb.exceptions_.MissingAttributeError: Instance of <class 'DB01048.drugbank_type'> lacks required attribute exported-on

Answer 1

一種更pythonic的方法可能是使用PyXB包來生成從xml文件解組的python對象。

• 安裝PyXB軟件包

pip install PyXB

• 從下載的xsd創建python模塊。 將創建DB01048.py

pyxbgen -m DB01048 -u drugbank-plus.xsd --no-validate-changes

• 使用生成的模塊作為

   from __future__ import print_function import DB01048 xml = open('DB01048.xml').read() d_01048 = DB01048.CreateFromDocument(xml) print(d_01048.drug[0].state) print(d_01048.drug[0].name) print(d_01048.drug[0].general_references.articles.article[0].pubmed_id) 

solid Abacavir 17356469

如果出現與某些exported-on屬性相關的MissingAttribute錯誤，請在DB01048.py的第一個Class之前添加以下行

pyxb.RequireValidWhenParsing(True) pyxb.RequireValidWhenParsing(False)