xml 到 dataframe 与 python 的严重嵌套

Question

我有一个嵌套很深的 xml，我正在尝试将其转换为数据框 object。

下面附上失败的尝试。

输入：johnny.xml 文件，包含以下文本-

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE collection SYSTEM "BioC.dtd">
<collection>
  <source/>
  <date/>
  <key/>
  <document>
    <id>2301222206</id>
    <infon key="tt_curatable">no</infon>
    <infon key="tt_version">1</infon>
    <infon key="tt_round">1</infon>
    <passage>
      <offset>0</offset>
      <text>Johnny likes pizza and chocolate, he lives in Italy with Emily.</text>
      <annotation id="1">
        <infon key="type">names</infon>
        <infon key="identifier">first_name</infon>
        <infon key="annotator">annotator_1</infon>
        <infon key="updated_at">2023-01-22T22:12:56Z</infon>
        <location offset="0" length="6"/>
        <text>Johnny</text>
      </annotation>
      <annotation id="3">
        <infon key="type">food</infon>
        <infon key="identifier"></infon>
        <infon key="annotator">annotator_2</infon>
        <infon key="updated_at">2023-01-22T22:13:51Z</infon>
        <location offset="13" length="19"/>
        <text>pizza and chocolate</text>
      </annotation>
      <annotation id="4">
        <infon key="type">location</infon>
        <infon key="identifier">europe</infon>
        <infon key="annotator">annotator_2</infon>
        <infon key="updated_at">2023-01-22T22:14:05Z</infon>
        <location offset="46" length="5"/>
        <text>Italy</text>
      </annotation>
      <annotation id="2">
        <infon key="type">names</infon>
        <infon key="identifier">first_name</infon>
        <infon key="annotator">annotator_1</infon>
        <infon key="updated_at">2023-01-22T22:13:08Z</infon>
        <location offset="57" length="5"/>
        <text>Emily</text>
      </annotation>
    </passage>
  </document>
</collection>

所需 output：

失败的尝试：

• 与 lxml

from lxml import objectify
root = objectify.parse('johnny.xml').getroot()
data=[]
for i in range(len(root.getchildren())):
    data.append([child.text for child in root.getchildren()[i].getchildren()])
df = pd.DataFrame(data)

结果 -

    0   1   2   3   4
0   None    None    None    None    None
1   None    None    None    None    None
2   None    None    None    None    None
3   2301222206  no  1   1   None

• 此解决方案使用递归 function（第二个答案）结果 -

id  infon-1 infon-2 infon-3 infon-key-1 infon-key-2 infon-key-3 passage-offset  passage-text    passage-annotation-id-1 ... passage-annotation-location-offset-3    passage-annotation-location-offset-4    passage-annotation-location-length-1    passage-annotation-location-length-2    passage-annotation-location-length-3    passage-annotation-location-length-4    passage-annotation-text-1   passage-annotation-text-2   passage-annotation-text-3   passage-annotation-text-4
0   NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
1   NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
2   NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
3   2301222206  no  1   1   tt_curatable    tt_version  tt_round    0   Johnny likes pizza and chocolate, he lives in Italy with Emily. 1   ... 46  57  6   19  5   5   Johnny  pizza and chocolate Italy   Emily

    4 rows × 56 columns

• 有同样的问题，首先使用 pandas_read_xml 库回答

import pandas_read_xml as pdx
p2 = 'Johnny.xml'
df = pdx.read_xml(p2, ['collection'])
df = pdx.fully_flatten(df)
df

结果生成了 47 行，这也不是我想要的。

• 也按照此处的建议尝试使用 beautifulsoup

谢谢！

Answer 1

这是一个使用pandas和xmltodict的例子

import pandas as pd
import xmltodict
from   pathlib import Path

xmldict = xmltodict.parse(Path("johnny.xml").read_text())

# unpack the names/text content from document.infon
xmldict["collection"]["document"]["infon"] = dict(
   list(row.values()) 
   for row in xmldict["collection"]["document"]["infon"]
)   

# unpack the names/text content from annotation dicts
xmldict["collection"]["document"]["passage"]["annotation"] = [
   { key: val for key, val in row.items() if key != "infon" } | 
   { col["@key"]: col.get("#text") for col in row["infon"] } 
   for row in xmldict["collection"]["document"]["passage"]["annotation"]
]

# use `.json_normalize()` to create a dataframe 
# `.explode()` turns each annotation into its own row
df = (
   pd.json_normalize(xmldict)
     .explode("collection.document.passage.annotation")
)

# remove annotations column 
# use `.json_normalize()` to create dataframe from annotation dicts
# concat/combine the columns with original dataframe
df = pd.concat(
   [
      df.drop(columns="collection.document.passage.annotation"),
      pd.json_normalize(df["collection.document.passage.annotation"])
        .set_index(df.index)
   ],
   axis=1
)

您可以根据需要重命名/删除列：

>>> df.columns
Index(['collection.source', 'collection.date', 'collection.key',
       'collection.document.id', 'collection.document.infon.tt_curatable',
       'collection.document.infon.tt_version',
       'collection.document.infon.tt_round',
       'collection.document.passage.offset',
       'collection.document.passage.text',
       '@id', 'text', 'type', 'identifier', 'annotator', 'updated_at', 
       'location.@offset', 'location.@length'],
      dtype='object')
>>> df[["@id", "text", "type", "identifier"]]
  @id                 text      type  identifier
0   1               Johnny     names  first_name
0   3  pizza and chocolate      food        None
0   4                Italy  location      europe
0   2                Emily     names  first_name

---

[更新]：

使用|的可能替代方法

for row in xmldict["collection"]["document"]["passage"]["annotation"]:
   row.update(
      { col["@key"]: col.get("#text") for col in row["infon"] } 
   )
   row.pop("infon", None)

发生的事情是row来自：

{'@id': '1', 
 'infon': [
    {'@key': 'type', '#text': 'names'}, 
    {'@key': 'identifier', '#text': 'first_name'}, 
    {'@key': 'annotator', '#text': 'annotator_1'}, 
    {'@key': 'updated_at', '#text': '2023-01-22T22:12:56Z'}],
 'location': {'@offset': '0', '@length': '6'}, 
 'text': 'Johnny'}

进入：

{'@id': '1', 
 'type': 'names',
 'identifier': 'first_name',
 'annotator': 'annotator_1',
 'updated_at': '2023-01-22T22:12:56Z',
 'location': {'@offset': '0', '@length': '6'}, 
 'text': 'Johnny'}

row["infon"]中的每个字典都被“解包”，键/文本值被“合并”到顶层。

然后infon密钥。

xmltodict使用@key / #text的原因是为了避免名称冲突。

如果在此示例中有内部{"@key": "text", ...} ，将其合并到顶层将覆盖现有的"text": "Johnny"

如果这是一个问题，您可以预先添加annotation. 键，所以你最终得到的是：

{'@id': '1',
 'annotation.type': 'names',
 'annotation.identifier': 'first_name',
 'annoation.annotator': 'annotator_1',
 'annotation.updated_at': '2023-01-22T22:12:56Z',
 'location': {'@offset': '0', '@length': '6'},
 'text': 'Johnny'}

这可能是我在最初的例子中应该做的。