Finding element in xml with python

Question

I am trying to parse XML before converting it's content into lists and then into CSV. Unfortunately, I think my search terms for finding the initial element are failing, causing subsequent searches further down the hierarchy. I am new to XML, so I've tried variations on namespace dictionaries and including the namespace references... The simplified XML is given below:

 <?xml version="1.0" encoding="utf-8"?> <StationList xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:add="http://www.govtalk.gov.uk/people/AddressAndPersonalDetails" xmlns:com="http://nationalrail.co.uk/xml/common" xsi:schemaLocation="http://internal.nationalrail.co.uk/xml/XsdSchemas/External/Version4.0/nre-station-v4-0.xsd" xmlns="http://nationalrail.co.uk/xml/station"> <Station xsi:schemaLocation="http://internal.nationalrail.co.uk/xml/XsdSchemas/External/Version4.0/nre-station-v4-0.xsd"> <ChangeHistory> <com:ChangedBy>spascos</com:ChangedBy> <com:LastChangedDate>2018-11-07T00:00:00.000Z</com:LastChangedDate> </ChangeHistory> <Name>Aber</Name> </Station>​ 

The Code I am using to try to extract the com/...xml/station / ChangedBy element is below

tree = ET.parse(rootfilepath + "NRE_Station_Dataset_2019_raw.xml")
root = tree.getroot()

#get at the tags and their data
#for elem in tree.iter():
#    print(f"this the tag {elem.tag} and this is the data: {elem.text}")

#open file for writing
station_data = open(rootfilepath + 'station_data.csv','w')

csvwriter = csv.writer(station_data)

station_head = []

count = 0
#inspiration for this code: http://blog.appliedinformaticsinc.com/how-to-  parse-and-convert-xml-to-csv-using-python/
#this is where it goes wrong; some combination of the namespace and the tag can't find anything in line 27, 'StationList'
for member in root.findall('{http://nationalrail.co.uk/xml/station}Station'):
station = []
if count == 0:
changedby = member.find('{http://nationalrail.co.uk/xml/common}ChangedBy').tag
station_head.append(changedby)

    name = member.find('{http://nationalrail.co.uk/xml/station}Name').tag
    station_head.append(name)

    count = count+1

changedby = member.find('{http://nationalrail.co.uk/xml/common}ChangedBy').text
station.append(changedby)

name = member.find('{http://nationalrail.co.uk/xml/station}Name').text
station.append(name)

csvwriter.writerow(station)

I have tried:

• using dictionaries of namespaces but that results in nothing being found at all
• using hard coded namespaces but that results in "Attribute Error: 'NoneType' object has no attribute 'tag'

Thanks in advance for all and any assistance.

Answer 1

Try lxml :

#!/usr/bin/env python3

from lxml import etree

ns = {"com": "http://nationalrail.co.uk/xml/common"}

with open("so.xml") as f:
    tree = etree.parse(f)
    for t in tree.xpath("//com:ChangedBy/text()", namespaces=ns):
        print(t)

Output:

spascos

Answer 2

You can use Beautifulsoup which is an html and xml parser

from bs4 import BeautifulSoup

fd = open(rootfilepath + "NRE_Station_Dataset_2019_raw.xml")  
soup = BeautifulSoup(fd,'lxml-xml')

for i in soup.findAll('ChangeHistory'):      
    print(i.ChangedBy.text)

Answer 3

First of all your XML is invalid ( </StationList> is absent at the end of a file).

Assuming you have valid XML file:

<?xml version="1.0" encoding="utf-8"?>
<StationList xmlns:xsd="http://www.w3.org/2001/XMLSchema"
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xmlns:add="http://www.govtalk.gov.uk/people/AddressAndPersonalDetails"
            xmlns:com="http://nationalrail.co.uk/xml/common"            xsi:schemaLocation="http://internal.nationalrail.co.uk/xml/XsdSchemas/External/Version4.0/nre-station-v4-0.xsd"
            xmlns="http://nationalrail.co.uk/xml/station">
  <Station xsi:schemaLocation="http://internal.nationalrail.co.uk/xml/XsdSchemas/External/Version4.0/nre-station-v4-0.xsd">
    <ChangeHistory>
      <com:ChangedBy>spascos</com:ChangedBy>
      <com:LastChangedDate>2018-11-07T00:00:00.000Z</com:LastChangedDate>
    </ChangeHistory>
    <Name>Aber</Name>
  </Station>​
</StationList>

Then you can convert your XML to JSON and simply address to the required value:

import xmltodict
with open('file.xml', 'r') as f:
    data = xmltodict.parse(f.read())
changed_by = data['StationList']['Station']['ChangeHistory']['com:ChangedBy']

Output:

spascos