XML 到 CSV Python 3.5.2

Question

I am trying to convert the following XML to CSV. The catch is every single entry might not have a value so it returns a NoneType . For example, in the XML shown below not every "entry" will have a "rule".

If this happens I would like the CSV file to contain either nothing or a generic value such as "EMPTY". I would like the CSV file to look something like this:

domain    serial     seqno   rule
 1       43434343434    1     21
 1       43434343434    1     21
 1       43434343434    1     EMPTY

By using the list comprehension shown below I was able to avoid the NoneType error. However, it appears that formatting the data in the CSV is where I need some assistance.

rows = [cleanhtml(str(entry))
        for entry in soup.find_all("entry")
        if entry.find(header_list[counter]) is not None]

#!/usr/bin/env python3

import csv
import re
from bs4 import BeautifulSoup

html_results='''<response status="success"><result>
      <job>
        <tenq>09:48:24</tenq>
        <tdeq>09:48:24</tdeq>
        <tlast>18:00:00</tlast>
        <status>FIN</status>
        <id>5955</id>
        <cached-logs>1118</cached-logs>
      </job>
      <log>
        <logs count="100" progress="100">
          <entry logid="4343">
            <domain>1</domain>
            <serial>43434343434</serial>
            <seqno>0</seqno>
            <actionflags>0x0</actionflags>
            <type>EXAMPLE</type>
            <subtype>EXAMPLE</subtype>
            <config_ver>0</config_ver>
            <src>1.1.1.1</src>
            <dst>1.1.1.1</dst>
            <rule>Rule 21</rule>
          </entry>
      <log>
          <entry logid="4343">
            <domain>1</domain>
            <serial>43434343434</serial>
            <seqno>0</seqno>
            <actionflags>0x0</actionflags>
            <type>EXAMPLE</type>
            <subtype>EXAMPLE</subtype>
            <config_ver>0</config_ver>
            <src>1.1.1.1</src>
            <dst>1.1.1.1</dst>
            <rule>Rule 21</rule>
          </entry>'''

def cleanhtml(raw_html):
  tags = re.compile('<.*?>')
  cleantext = re.sub(tags, '', raw_html)
  return cleantext

soup = BeautifulSoup(html_results, 'html.parser')

header_list = ['domain',"serial","seqno","actionflags","type","subtype","config_ver","src","dst","rule"]

query_results = open("query_results.csv","w")

csvwriter = csv.writer(query_results)

csvwriter.writerow(header_list)

num_of_logs = soup.find("logs").get("count")

counter = 0

rows = [cleanhtml(str(entry)) for entry in soup.find_all("entry") if entry.find(header_list[counter]) is not None]

csvwriter.writerows(rows)

query_results.close()

Answer 1

You are not handling your entry subelements; you are merely turning each entry to text and removing the XML tag markup. You need to produce a list or dictionary with each subelement entry teased out separately.

If you produce a dictionary of the nested elements, then thecsv.DictWriter() class could handle filling in empty columns for you, without additional coding:

def entry_to_dict(entry):
    return {tag.name: tag.get_text() for tag in entry.find_all()}

header_list = ['domain', 'serial', 'seqno', 'actionflags', 'type', 'subtype', 'config_ver', 'src', 'dst', 'rule']

soup = BeautifulSoup(html_results, 'html.parser')
with open("query_results.csv","w") as query_results:
    csvwriter = csv.DictWriter(query_results, header_list, restval='EMPTY')
    csvwriter.writeheader()
    csvwriter.writerows(entry_to_dict(entry) for entry in soup.find_all('entry'))

Here, the restval argument tells the writer how to handle missing values in each row. header_list is passed in as the field names, so the writer knows what keys to expect in each row dictionary.

entry_to_dict() simply turns each nested element in an entry into a key-value pair in a dictionary, with thetag.get_text() function doing the work of turning element contents into text. The

For your demo XML data, this produces:

>>> import sys
>>> csvwriter = csv.DictWriter(sys.stdout, header_list, restval='EMPTY')
>>> csvwriter.writeheader()
domain,serial,seqno,actionflags,type,subtype,config_ver,src,dst,rule
>>> csvwriter.writerows(entry_to_dict(entry) for entry in soup.find_all('entry'))
1,43434343434,0,0x0,EXAMPLE,EXAMPLE,0,1.1.1.1,1.1.1.1,Rule 21
1,43434343434,0,0x0,EXAMPLE,EXAMPLE,0,1.1.1.1,1.1.1.1,Rule 21

This doesn't actually contain any empty elements, but when I add some, you can see that EMPTY is used to fill those in:

>>> html_results += '''</log><log>
...           <entry logid="4343">
...             <domain>1</domain>
...             <serial>43434343434</serial>
...             <seqno>0</seqno>
...             <actionflags>0x0</actionflags>
...             <type>EXAMPLE</type>
...             <subtype>EXAMPLE</subtype>
...             <!-- incomplete entry, config_ver, src, dst and rule missing -->
...          </entry>
...        </log>'''
>>> soup = BeautifulSoup(html_results, 'html.parser')
>>> csvwriter = csv.DictWriter(sys.stdout, header_list, restval='EMPTY')
>>> csvwriter.writeheader()
domain,serial,seqno,actionflags,type,subtype,config_ver,src,dst,rule
>>> csvwriter.writerows(entry_to_dict(entry) for entry in soup.find_all('entry'))
1,43434343434,0,0x0,EXAMPLE,EXAMPLE,0,1.1.1.1,1.1.1.1,Rule 21
1,43434343434,0,0x0,EXAMPLE,EXAMPLE,0,1.1.1.1,1.1.1.1,Rule 21
1,43434343434,0,0x0,EXAMPLE,EXAMPLE,EMPTY,EMPTY,EMPTY,EMPTY

As a final note: consider installing the lxml library , and use the xml parser in BeautifulSoup:

soup = BeautifulSoup(html_results, 'xml')

This ensures that your XML is parsed correctly at all times (the HTML parser is fault-tolerant in ways that XML should not be, and is case-insensitive, which could cause issues with mixed-case XML data).