I have an XML file that I need to extract data from and insert into a database table. My struggle is that the XML data structure could contain inconsistant child tags. Meaning that (in the example below) one parent <Field>
tag, may or may not contain a <ListValue>
tag.
This is a short example and I will be adding additional <Field>
tags potentially containing another <ListValue>
tag. Note: All <Field>
tags are expected to remain at the same level below the <Record>
tag.
I want to see if anyone has a more "pythonic" way of converting this data than my example below. Maybe with list comprehension?
I will need to insert up to 4,000,000 <Record>
level rows of data into a database, so I don't want to waste more time looping through the XML than is necessary. Speed will be essential.
Any assistance will be appreciated.
<?xml version="1.0" encoding="utf-16"?>
<Records count="10">
<Metadata>
<FieldDefinitions>
<FieldDefinition id="15084" guid="f3426157-cbcb-4293-94e5-9f1c993db4b5" name="CCR_ID" alias="CCR_ID" />
<FieldDefinition id="16335" guid="5dfddb49-9a7a-46ee-9bd2-d5bbed97a48d" name="Coming Due" alias="Coming_Due" />
</FieldDefinitions>
</Metadata>
<LevelCounts>
<LevelCount id="35" guid="661c747f-7ce5-474a-b320-044aaec7a5b1" count="10" />
</LevelCounts>
<Record contentId="20196771" levelId="35" levelGuid="661c747f-7ce5-474a-b320-044aaec7a5b1" moduleId="265" parentId="0">
<Field id="15084" guid="f3426157-cbcb-4293-94e5-9f1c993db4b5" type="1">100383-320-V0217111</Field>
<Field id="16335" guid="5dfddb49-9a7a-46ee-9bd2-d5bbed97a48d" type="4">
<ListValues>
<ListValue id="136572" displayName="121 - 180 days out">121 - 180 days out</ListValue>
</ListValues>
</Field>
</Record>
<Record contentId="20205193" levelId="35" levelGuid="661c747f-7ce5-474a-b320-044aaec7a5b1" moduleId="265" parentId="0">
<Field id="15084" guid="f3426157-cbcb-4293-94e5-9f1c993db4b5" type="1">100383-320-V0217267</Field>
<Field id="16335" guid="5dfddb49-9a7a-46ee-9bd2-d5bbed97a48d" type="4">
<ListValues>
<ListValue id="136572" displayName="121 - 180 days out">121 - 180 days out</ListValue>
</ListValues>
</Field>
</Record>
<Record contentId="20196779" levelId="35" levelGuid="661c747f-7ce5-474a-b320-044aaec7a5b1" moduleId="265" parentId="0">
<Field id="15084" guid="f3426157-cbcb-4293-94e5-9f1c993db4b5" type="1">100384-320-V0217111</Field>
<Field id="16335" guid="5dfddb49-9a7a-46ee-9bd2-d5bbed97a48d" type="4">
<ListValues>
<ListValue id="136572" displayName="121 - 180 days out">121 - 180 days out</ListValue>
</ListValues>
</Field>
</Record>
</Records>
Here is my code for parsing the data:
from xml.etree import ElementTree
import pandas as pd
xml_string = '''SEE STRING ABOVE'''
auth_token = ElementTree.fromstring(xml_string.text)
dct = []
cols = ['CCR_ID', 'Coming_Due']
for r in auth_token.findall("Record"):
for f in r.findall("Field"):
if f.attrib['id'] == '15084':
ccr_id = f.text
for l in f.findall(".//ListValue"):
coming_due = l.text
dct.append((ccr_id, coming_due))
df = pd.DataFrame(dct)
df.columns = cols
print(df)
Here are my results:
CCR_ID Coming_Due
0 100383-320-V0217111 121 - 180 days out
1 100383-320-V0217267 121 - 180 days out
2 100384-320-V0217111 121 - 180 days out
3 100384-320-V0217267 121 - 180 days out
4 100681-320-V0217111 121 - 180 days out
5 100681-320-V0217267 11 - 30 days out
6 100684-320-V0217111 121 - 180 days out
7 100684-320-V0217267 11 - 30 days out
8 100685-320-V0217111 121 - 180 days out
9 100685-320-V0217267 11 - 30 days out
If I understand you correctly, using pandas read_xml()
may help:
df = pd.read_xml(string,"//Record//*")
df2= df[['Field','displayName']].copy()
df2['displayName'] = df2['displayName'].shift(-3)
df2.set_axis(['CCR_ID', 'Coming_Due'], axis=1,inplace=True)
df2.dropna()
Output based on your sample xml:
Field displayName
0 100383-320-V0217111 121 - 180 days out
4 100383-320-V0217267 121 - 180 days out
8 100384-320-V0217111 121 - 180 days out
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.