使用lxml解析奇怪的结构化XML

Question

I have a number of XML files that I need to parse. I've written some code that works, but is ugly, and I'd like to get some advice from people more experienced with XML than I am.

First of all, I might be using some terms in the wrong context, because my experience with XML is limited. By element, unless specified otherwise, I mean something like this:

 <root>
  <element>
   ...
  </element>
  <element>
   ...
  </element>
 </root>  

Anyway, each file consist of a number of elements, with a number of child elements (obviously). What stumps me is that the relevant values need to be accessed in four different ways;

1) Node text:

<tag>value</tag>

2) Attribute:

<tag attribute="value"></tag>

3) A value "hidden" inside a tag ("true" in this case):

<tag><boolean.true/></tag>

4) Values inside tags of the same name ("tagA"), but with "grandparent" tags with different names ("tag1" and "tag2"), all within the same element. "tagA" is of no use to me, instead I will be looking for "tag1" and "tag2".

<element>
   <tag1><tagA>value</tagA><tag1>
   <tag2><tagA>value</tagA></tag2>
</element>

At the moment I have a dictionary with each file as a key. The values are dictionaries with the keys "attribute", "node text", "tag" and "parent element".

Example:

{'file1.xml' : 'attributes' : {'Person': 'Id', 'Car' : 'Color'},
               'node text': ['Name', 'Address'],
}

Where "Person" and "Car" are tags, and "Id" and "Color" are attribute names.

This makes it easy to iterate over all elements and inspect each tag, and if there is a match in the dictionary (if elem.tag in dict['file1.xml']['attributes']), extract the value.

So as I said, the code works, but I don't like my solution. Also, not all the elements have all the child elements (for example, a Person might not own a car, then that tag will be missing altogether), and I need to give assign those values "None". Right now I get all the tags that should exist for every element in each file, turn them into a set, then check the difference between those and the set of tags that I've actually extracted values from for that element. Again, the code is pretty ugly.

Hopefully this mess makes some sense.

edit:

I used JF Sebastian's suggestion of storing the xpath to each value in a dictionary with the field name as the key and xpath as value.

Answer 1

You could streamline your input code by using xpath expressions relative your element instead of a complex data-structure eg, #1-4 cases:

1. tag/text()
2. tag/@attribute
3. name(DTBoolean/*[1])
4. (tag1|tag2)/*/text()

What output data-structure to use depends on how do you like it to be used in your code later. You could start with a structure that is most convenient for your current code. And evolve it to a more general solution later when you better understand the requirements.

I output it to csv, where each element is one row in the csv file. ... I use a defaultdict to store the elements and then store those in a list before I output them to csv.

You could use ordinary dict and csv.DictWriter(fieldnames=xpathdict.keys()):

# for each element
row_dict = dict.fromkeys(xpathdict.keys())
...
# for each key 
row_dict[key] = element.xpath(xpathdict[key]) or None
...
dictwriter.writerow(row_dict)

Where xpathdict is a mapping between field names and corresponding xpath expressions. For generality you could store function objects f(element) -> csv field instead of/in addition to xpath exprs.

Answer 2

I don't think #3 is legal XML because there's no opening tag associated with and even if it's somewhere else, it wouldn't be properly nested in that example. The expression will be interpreted as a closing tag because of the < character.

Answer 3

I'm assuming that you'd want to take something like this:

<root>
  <element>
    <text_attribute>Some Text</text_attribute>
    <attribute var="blah"/>
    <bool_attribute><boolean.true/></bool_attribute>
  </element>
  <element>
    <text_attribute>Some more Text</text_attribute>
    <attribute var="blah again"/>
    <bool_attribute><boolean.false/></bool_attribute>
  </element>
</root>

And get something like this:

[
   { "text_attribute":"Some Text", "attribute":"blah", "bool_attribute":True },
   { "text_attribute":"Some more Text", "attribute":"blah again", "bool_attribute":False }
]

To do this I'd do something like this (untested):

# Helper function so we can extract a default from an xpath result if empty
def get_first(x, default_value):
  if(len(x)>0) return x[0]
  return default_value

# Parse one element
def process_element( e ):
  retval = {}
  retval['text_attribute'] = get_first(e.xpath("text_attribute/text()"), "default text")
  retval['attribute'] = get_first( e.xpath("attribute/@var"), "default attribute")
  retval['bool_attribute'] = get_first( e.xpath("bool_attribute/boolean.true"), False )
  return retval

# Parse all the elements
elements = []
elements_xml = xml.xpath('/root/element')
for e in elements_xml:
  elements.push( process_element(e) )