使用 R 从大型 XML 中提取数据

Question

. Answers to this question are eligible for a +50 reputation bounty. MLEN is looking for an to this question.

I have a large XML that I cant parse completely in R due to memory shortage. I would like just to extract some specific columns. I found other asked similar questions:

How to read large (~20 GB) xml file in R? Storing specific XML node values with R's xmlEventParse

I cant get it to work though with my data, it runs, but no data is returned. I did try to adjust the suggested solutions to my XML but it still does not work. Might be my lack of knowledge XML. Below is a example of my XML data, where cl, clssc, clp, clpssc, primclp are the columns. How can I extract only cl and clssc without parsing the whole document first?

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<abc:abc xmlns:abc="http://abc/abc" xsi:schemaLocation="http://abc/abc lala_20Q2.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <chcp>
    <cl>2000000</cl>
    <clssc>10934</clssc>
    <clp>200000</clp>
    <clpssc>10934</clpssc>
    <primclp>Y</primclp>
  </chcp>
  <chcp>
    <cl>2000000</cl>
    <clssc>10934</clssc>
    <clp>200000</clp>
    <clpssc>10934</clpssc>
    <primclp>Y</primclp>
  </chcp>
  <chcp>
    <cl>2000000</cl>
    <clssc>10934</clssc>
    <clp>2000000</clp>
    <clpssc>10934</clpssc>
    <primclp>Y</primclp>
  </chcp>
</abc:abc>

Answer 1

The disk.frame package is made to handle medium sized data. It enables batch conversion of data into .fst and speedy I/O through the fst package , and fast data manipulation using data.table . Here, the dtplyr interface to the data.table package is used for the final wrangling.

Step 1: preparing the input file*

1.1 Create a folder and place your .xml file in there.

1.2 Remove the first two lines, and the last line so that you are left with a .xml file that has this structure:

  <chcp>
    <cl>2000000</cl>
    <clssc>10934</clssc>
    <clp>200000</clp>
    <clpssc>10934</clpssc>
    <primclp>Y</primclp>
  </chcp>
  <chcp>
    <cl>2000000</cl>
    <clssc>10934</clssc>
    <clp>200000</clp>
    <clpssc>10934</clpssc>
    <primclp>Y</primclp>
  </chcp>
  <chcp>
    <cl>2000000</cl>
    <clssc>10934</clssc>
    <clp>2000000</clp>
    <clpssc>10934</clpssc>
    <primclp>Y</primclp>
  </chcp>

Step 2: setup disk.frame

library(tidyverse)
library(disk.frame)

path <- file.path(file.choose()) # filepath to your folder containing .fst
setup_disk.frame(workers = 10) # adjust this to your machine
options(future.globals.maxSize = Inf)
old <- getOption("scipen") ; options(scipen = 100) # prevent scientific numbers later

Step 3: convert .xml to .fst

l <- csv_to_disk.frame( #works for .xml
  paste0(path, "b.xml"), # replace b.xml with your filename (in folder)
  outdir = paste0(path, "combined.df"),
  in_chunk_size = 7,
  backend = "data.table", header = F)

This gives object l of class "disk.frame" "disk.frame.folder" in your R environment. You should now have a subfolder "combined.df" which contains a bunch of .fst files in your specified directory.

Step 4: read in the required columns & tidy data

cbind(
  get_chunk(l, 1) %>%
    `[`(seq(2, nrow(.), 2)),  
  get_chunk(l, 2) %>%
    `[`(seq(1, nrow(.), 2))
) %>%
  rename("cl" = 1, "clssc" = 2) %>%
  mutate(across(.fns = parse_number)) %>%
  as_tibble() # omit this to keep data.table

# A tibble: 3 x 2
       cl clssc
    <dbl> <dbl>
1 2000000 10934
2 2000000 10934
3 2000000 10934

Finally, don't forget to revert the scientific notation options options(scipen = old) .

* Note: Step 1 can likely be avoided by playing with the chunk sizes or through some manipulation of the .xml from within R. This I don't know how to do (yet).

Note2: Recommend to carefully read the disk.frame documentation for tips on how to setup properly for your machine.

Answer 2

on a windows machine

Here ia an approach where you use the output from the windows findstr -command to import data using data.table::fread() . It filters the data using the windows-version of 'grep' before it is loaded into R. This way you will not run into memory problems very soon.

location of the xml: e:/testdata.xml

further explanation is in data's comments below

library(data.table)
# Import output from windows findstr-command
#  assumes location of data is e:/testdata.xml
#  !! use \\ in path, else findstr does not undeerstand !!
DT <- data.table::fread(cmd = 'findstr "<clssc> <cl>" e:\\testdata.xml', 
                        sep = "\n", col.names = "line", header = FALSE )
#                    line
# 1:     <cl>2000000</cl>
# 2: <clssc>10934</clssc>
# 3:     <cl>2000000</cl>
# 4: <clssc>10934</clssc>
# 5:     <cl>2000000</cl>
# 6: <clssc>10934</clssc>

# Extract data from raw line
DT[, name  := gsub("^<(.+?)>.*$", "\\1", line)]
DT[, value := gsub("^.*>([0-9]+?)<.*$", "\\1", line)]
#                    line  name   value
# 1:     <cl>2000000</cl>    cl 2000000
# 2: <clssc>10934</clssc> clssc   10934
# 3:     <cl>2000000</cl>    cl 2000000
# 4: <clssc>10934</clssc> clssc   10934
# 5:     <cl>2000000</cl>    cl 2000000
# 6: <clssc>10934</clssc> clssc   10934

# Build some id's
DT[, id := rowid(name)]

# Cast to wide format
dcast(DT, id ~ name, value.var = "value")

#    id      cl clssc
# 1:  1 2000000 10934
# 2:  2 2000000 10934
# 3:  3 2000000 10934

on a unix machine
I cannot test, since I only use windows at this location.
Replace the cmd='...' part from findstr-command (and the regex) with the grep-command of your system.