使用 Python 解析和查询大 XML 文件

Question

I am working on a project for which I have to parse and query a relatively large xml file in python. I am using a dataset with data about scientific articles. The dataset can be found via this link ( https://dblp.uni-trier.de/xml/dblp.xml.gz ). There are 7 types of entries in the dataset: article , inproceedings , proceedings , book , incollection , phdthesis and masterthesis . An entry has the following attributes: author , title , year and either journal or booktitle .

I am looking for the best way to parse this and consequently perform queries on the dataset. Examples of queries that I would like to perform are:

• retrieve articles that have a certain author
• retrieve articles if the title contains a certain word
• retrieve articles to which author x and author y both contributed.
• ...

Herewith a snapshot of an entry in the xml file:

<article mdate="2020-06-25" key="tr/meltdown/s18" publtype="informal">
<author>Paul Kocher</author>
<author>Daniel Genkin</author>
<author>Daniel Gruss</author>
<author>Werner Haas 0004</author>
<author>Mike Hamburg</author>
<author>Moritz Lipp</author>
<author>Stefan Mangard</author>
<author>Thomas Prescher 0002</author>
<author>Michael Schwarz 0001</author>
<author>Yuval Yarom</author>
<title>Spectre Attacks: Exploiting Speculative Execution.</title>
<journal>meltdownattack.com</journal>
<year>2018</year>
<ee type="oa">https://spectreattack.com/spectre.pdf</ee>
</article> 

Does anybody have an idea on how to do to this efficiently?

I have experimented with using the ElementTree. However, when parsing the file I get the following error:

xml.etree.ElementTree.ParseError: undefined entity Ö: line 90, column 17

Additionally, I am not sure if using the ElementTree will be the most efficient way for querying this xml file.

Answer 1

If the file is large, and you want to perform multiple queries, then you don't want to be parsing the file and building a tree in memory every time you do a query. You also don't want to be writing the queries in low-level Python, you need a proper query language.

You should be loading the data into an XML database such as BaseX or ExistDB. You can then query it using XQuery. This will be a bit more effort to set up, but will make your life a lot easier in the long run.