使用xmltree解析大型python xml

Question

I have a python script that parses huge xml files ( largest one is 446 MB)

    try:
        parser = etree.XMLParser(encoding='utf-8')
        tree = etree.parse(os.path.join(srcDir, fileName), parser)
        root = tree.getroot()
    except Exception, e:
        print "Error parsing file "+str(fileName) + " Reason "+str(e.message)

    for child in root:
        if "PersonName" in child.tag:
            personName = child.text

This is what my xml looks like :

<?xml version="1.0" encoding="utf-8"?>
<MyRoot xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" uuid="ertr" xmlns="http://www.example.org/yml/data/litsmlv2">
  <Aliases authority="OPP" xmlns="http://www.example.org/yml/data/commonv2">
     <Description>myData</Description>
     <Identifier>43hhjh87n4nm</Identifier>
  </Aliases>
  <RollNo uom="kPa">39979172.201167159</RollNo>
  <PersonName>Miracle Smith</PersonName>
  <Date>2017-06-02T01:10:32-05:00</Date>
....

All I want to do is get the PersonName tags contents thats all. Other tags I don't care about.

Sadly My files are huge and I keep getting this error when I use the code above :

Error parsing file 2eb6d894-0775-e611.xml Reason unknown error, line 1, column 310915857
Error parsing file 2ecc18b5-ef41-e711-80f.xml Reason Extra content at the end of the document, line 1, column 3428182
Error parsing file 2f0d6926-b602-e711-80f4-005.xml Reason Extra content at the end of the document, line 1, column 6162118
Error parsing file 2f12636b-b2f5-e611-80f3-00.xml Reason Extra content at the end of the document, line 1, column 8014679
Error parsing file 2f14e35a-d22b-4504-8866-.xml Reason Extra content at the end of the document, line 1, column 8411238
Error parsing file 2f50c2eb-55c6-e611-80f0-005056a.xml Reason Extra content at the end of the document, line 1, column 7636614
Error parsing file 3a1a3806-b6af-e611-80ef-00505.xml Reason Extra content at the end of the document, line 1, column 11032486

My XML is perfectly fine and has no extra content .Seems that the large files parsing causes the error. I have looked at iterparse() but it seems to complex for what I want to achieve as it provides parsing of the whole DOM while I just want that one tag that is under the root. Also , does not give me a good sample to get the correct value by tag name ?

Should I use a regex parse or grep /awk way to do this ? Or any tweak to my code will let me get the Person name in these huge files ?

UPDATE: Tried this sample and it seems to be printing the whole world from the xml except my tag ?

Does iterparse read from bottom to top of file ? In that case it will take a long time to get to the top ie my PersonName Tag ? I tried changing the line below to read end to start events=("end", "start") and it does the same thing !!!

path = []
for event, elem in ET.iterparse('D:\\mystage\\2-80ea-005056.xml', events=("start", "end")):
    if event == 'start':
            path.append(elem.tag)
    elif event == 'end':
            # process the tag
            print elem.text  // prints whole world 
            if elem.tag == 'PersonName':
                print elem.text
            path.pop()

Answer 1

Iterparse is not that difficult to use in this case.

temp.xml is the file presented in your question with a </MyRoot> stuck on as a line at the end.

Think of the source = as boilerplace, if you will, that parses the xml file and returns chunks of it element-by-element, indicating whether the chunk is the 'start' of an element or the 'end' and supplying information about the element.

In this case we need consider only the 'start' events. We watch for the 'PersonName' tags and pick up their texts. Having found the one and only such item in the xml file we abandon the processing.

>>> from xml.etree import ElementTree
>>> source = iter(ElementTree.iterparse('temp.xml', events=('start', 'end')))
>>> for an_event, an_element in source:
...     if an_event=='start' and an_element.tag.endswith('PersonName'):
...         an_element.text
...         break
... 
'Miracle Smith'

Edit, in response to question in a comment:

Normally you wouldn't do this since iterparse is intended for use with large chunks of xml. However, by wrapping a string in a StringIO object it can be processed with iterparse .

>>> from xml.etree import ElementTree
>>> from io import StringIO
>>> xml = StringIO('''\
... <?xml version="1.0" encoding="utf-8"?>
... <MyRoot xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" uuid="ertr" xmlns="http://www.example.org/yml/data/litsmlv2">
...   <Aliases authority="OPP" xmlns="http://www.example.org/yml/data/commonv2">
...        <Description>myData</Description>
...             <Identifier>43hhjh87n4nm</Identifier>
...               </Aliases>
...                 <RollNo uom="kPa">39979172.201167159</RollNo>
...                   <PersonName>Miracle Smith</PersonName>
...                     <Date>2017-06-02T01:10:32-05:00</Date>
... </MyRoot>''')
>>> source = iter(ElementTree.iterparse(xml, events=('start', 'end')))
>>> for an_event, an_element in source:
...     if an_event=='start' and an_element.tag.endswith('PersonName'):
...         an_element.text
...         break
...     
'Miracle Smith'