堆栈内存溢出
  繁体   English   中英

Nokogiri :: XML :: Reader - 处理大型XML文件并跳过不感兴趣的节点

[英]Nokogiri::XML::Reader - processing large XML files and skipping nodes of no interest

 原文  2017-11-13 16:17:27  0   2    ruby/ xml/ nokogiri/ sax

问题描述

我有一些xml的格式如下所示，我正在尝试使用Nokogiri :: XML :: Reader解析，因为文件大小非常大（~1GB）。 该文件包含许多以下格式的packets 。

从每个packet我需要收集frame.time_epoch ， s1ap.procedureCode 。

我目前正在做以下事情。 

data = []
file = `some_file.xml`
reader = Nokogiri::XML::Reader(File.open(file))
reader.each do |node|
    if (node.name == 'packet' && node.node_type == Nokogiri::XML::Reader::TYPE_ELEMENT)
      doc = Nokogiri::XML(node.outer_xml)
      next if !doc.css("field[name='s1ap.procedureCode']") ## do nothing if the <packet> is not of s1ap type
      epochTime = doc.css("field[name='frame.time_epoch']").first["show"].to_i
      procedureCode = procedureCode_node = doc.css("field[name='s1ap.procedureCode']").first["show"].to_i
      data << { epochTime: epochTime, procedureCode: procedureCode }
    end
end

问题

我面临的挑战是解析非常缓慢。 一两件事我注意到的是，读取器扫描所有后续行一个内<packet> </packet> -有一种方法可以有读取器移动到下一个节点与名称作为packet而通过每一行一个内去packet进一步。

XML格式 

<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet type="text/xsl" href="pdml2html.xsl"?>
<packet>
  <proto name="geninfo" pos="0" showname="General information" size="126">
    <field name="num" pos="0" show="6" showname="Number" value="6" size="126"/>
  </proto>
  <proto name="frame" showname="Frame 6: 126 bytes on wire (1008 bits), 126 bytes captured (1008 bits) on interface 0" size="126" pos="0">
    <field name="frame.encap_type" showname="Encapsulation type: Ethernet (1)" size="0" pos="0" show="1"/>
    <field name="frame.time_epoch" showname="Epoch Time: 1474267259.184197000 seconds" size="0" pos="0" show="1474267259.184197000"/>
  </proto>
  <proto name="eth" showname="Ethernet II, Src: JuniperN_e6:a6:cc (40:b4:f0:e6:a6:cc), Dst: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="14" pos="0">
    <field name="eth.dst" showname="Destination: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="6" pos="0" show="ac:16:2d:89:a5:91" value="ac162d89a591">
      <field name="eth.dst_resolved" showname="Destination (resolved): HewlettP_89:a5:91" hide="yes" size="6" pos="0" show="HewlettP_89:a5:91" value="ac162d89a591"/>
    </field>
    <field name="eth.type" showname="Type: IPv4 (0x0800)" size="2" pos="12" show="0x00000800" value="0800"/>
  </proto>  
  <proto name="s1ap" showname="S1 Application Protocol" size="45" pos="78">
    <field name="per.choice_index" showname="Choice Index: 0" hide="yes" size="1" pos="78" show="0" value="00"/>
    <field name="s1ap.S1AP_PDU" showname="S1AP-PDU: initiatingMessage (0)" size="45" pos="78" show="0" value="000b402900000300000005c007c03ae900080003403b53001a0012113743f99f9500075d010605f070c04070c1">
      <field name="s1ap.initiatingMessage_element" showname="initiatingMessage" size="45" pos="78" show="" value="">
        <field name="s1ap.procedureCode" showname="procedureCode: id-downlinkNASTransport (11)" size="1" pos="79" show="11" value="0b"/>
       </field>
    </field>
  </proto>
</packet>
<packet>
  <proto name="geninfo" pos="0" showname="General information" size="126">
    <field name="num" pos="0" show="6" showname="Number" value="6" size="126"/>
  </proto>
  <proto name="frame" showname="Frame 6: 126 bytes on wire (1008 bits), 126 bytes captured (1008 bits) on interface 0" size="126" pos="0">
    <field name="frame.encap_type" showname="Encapsulation type: Ethernet (1)" size="0" pos="0" show="1"/>
    <field name="frame.time_epoch" showname="Epoch Time: 1474267260.184197000 seconds" size="0" pos="0" show="1474267259.184197000"/>
  </proto>
  <proto name="eth" showname="Ethernet II, Src: JuniperN_e6:a6:cc (40:b4:f0:e6:a6:cc), Dst: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="14" pos="0">
    <field name="eth.dst" showname="Destination: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="6" pos="0" show="ac:16:2d:89:a5:91" value="ac162d89a591">
      <field name="eth.dst_resolved" showname="Destination (resolved): HewlettP_89:a5:91" hide="yes" size="6" pos="0" show="HewlettP_89:a5:91" value="ac162d89a591"/>
    </field>
    <field name="eth.type" showname="Type: IPv4 (0x0800)" size="2" pos="12" show="0x00000800" value="0800"/>
  </proto>  
  <proto name="s1ap" showname="Some other protocol" size="45" pos="78">
    <field name="per.choice_index" showname="Choice Index: 0" hide="yes" size="1" pos="78" show="0" value="00"/>
    <field name="other.OTH_PDU" showname="S1AP-PDU: initiatingMessage (0)" size="45" pos="78" show="0" value="000b402900000300000005c007c03ae900080003403b53001a0012113743f99f9500075d010605f070c04070c1">
      <field name="other.initiatingMessage_element" showname="initiatingMessage" size="45" pos="78" show="" value="">
        <field name="other.procedureCode" showname="procedureCode: id-someTransport (99)" size="1" pos="79" show="11" value="0b"/>
       </field>
    </field>
  </proto>
</packet>
<packet>
  <proto name="geninfo" pos="0" showname="General information" size="126">
    <field name="num" pos="0" show="6" showname="Number" value="6" size="126"/>
  </proto>
  <proto name="frame" showname="Frame 6: 126 bytes on wire (1008 bits), 126 bytes captured (1008 bits) on interface 0" size="126" pos="0">
    <field name="frame.encap_type" showname="Encapsulation type: Ethernet (1)" size="0" pos="0" show="1"/>
    <field name="frame.time_epoch" showname="Epoch Time: 1474267261.184197000 seconds" size="0" pos="0" show="1474267259.184197000"/>
  </proto>
  <proto name="eth" showname="Ethernet II, Src: JuniperN_e6:a6:cc (40:b4:f0:e6:a6:cc), Dst: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="14" pos="0">
    <field name="eth.dst" showname="Destination: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="6" pos="0" show="ac:16:2d:89:a5:91" value="ac162d89a591">
      <field name="eth.dst_resolved" showname="Destination (resolved): HewlettP_89:a5:91" hide="yes" size="6" pos="0" show="HewlettP_89:a5:91" value="ac162d89a591"/>
    </field>
    <field name="eth.type" showname="Type: IPv4 (0x0800)" size="2" pos="12" show="0x00000800" value="0800"/>
  </proto>  
  <proto name="s1ap" showname="S1 Application Protocol" size="45" pos="78">
    <field name="per.choice_index" showname="Choice Index: 0" hide="yes" size="1" pos="78" show="0" value="00"/>
    <field name="s1ap.S1AP_PDU" showname="S1AP-PDU: initiatingMessage (0)" size="45" pos="78" show="0" value="000b402900000300000005c007c03ae900080003403b53001a0012113743f99f9500075d010605f070c04070c1">
      <field name="s1ap.initiatingMessage_element" showname="initiatingMessage" size="45" pos="78" show="" value="">
        <field name="s1ap.procedureCode" showname="procedureCode: id-uplinkTransport (13)" size="1" pos="79" show="13" value="0b"/>
       </field>
    </field>
  </proto>
</packet>
<!-- more <packet>s here -->

2 个解决方案

解决方案1
 3  2017-11-21 04:29:05

对于如此庞大的文档，您应该使用SAX解析器

http://www.rubydoc.info/github/sparklemotion/nokogiri/Nokogiri/XML/SAX

流处理大型文档而不将整个内容拉入内存并作为DOM进行解析是可取的。 特别是考虑到问题只需要一次通过。

以下是通过使用SAX流式传输XML来完成任务的代码： 

require 'nokogiri'

class PacketFilter < Nokogiri::XML::SAX::Document
  def initialize
    reset
  end

  def end_document
    puts 'the document has ended'
  end

  def start_element(name, attributes = [])
    case name
    when 'packet'
      @in_packet = true
    when 'proto'
      @have_s1ap = @in_packet && attribute_value(attributes, 'name') == 's1ap'
    when 'field'
      case attribute_value(attributes, 'name')
      when 's1ap.procedureCode'
        @procedure_code = attribute_value(attributes, 'showname')
      when 'frame.time_epoch'
        @epoch_time = attribute_value(attributes, 'showname')
      end
    end
  end

  def end_element(name)
    if name == 'packet'
      puts "#{@procedure_code}, #{@epoch_time}" if @have_s1ap
      reset
    end
  end

  private

  def attribute_value(attributes, name)
    attributes.reduce(nil) do |value, assoc|
      assoc[0] == name ? assoc[1] : value
    end
  end

  def reset
    @in_packet = false
    @have_s1ap = false
    @procedure_code = nil
    @epoch_time = nil
  end
end

parser = Nokogiri::XML::SAX::Parser.new(PacketFilter.new)
parser.parse($stdin)

如果将数据样本粘贴到data.xml ，将上面的ruby粘贴到slap.rb ： 

$ cat data.xml | ruby poke.rb
procedureCode: id-downlinkNASTransport (11), Epoch Time: 1474267259.184197000 seconds
the document has ended

解决方案2
 0  2017-11-21 18:30:13

您可以只遍历packet元素，而不是遍历每个节点，然后跳过任何不符合您标准的packet元素。 这将只执行packet元素而不是所有元素，这应该明显更快。 

data = []
file = 'some_file.xml'
doc = Nokogiri::XML.fragment(File.read(file)) # use `read` instead of `open`
doc.xpath('packet').each do |packet|
    next if !packet.css("field[name='s1ap.procedureCode']") ## do nothing if the <packet> is not of s1ap type
    epochTime = packet.css("field[name='frame.time_epoch']").first["show"].to_i
    procedureCode = procedureCode_node = packet.css("field[name='s1ap.procedureCode']").first["show"].to_i
    data << { epochTime: epochTime, procedureCode: procedureCode }
end 

» data
=> [{:epochTime=>1474267259, :procedureCode=>11}

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 如何使用 Nokogiri::XML::Reader 解析大型 XML 文件？ 使用 Ruby 和 Nokogiri 解析大型 XML 文件 Nokogiri :: XML :: Reader跳过名称空间 使用Nokogiri解析大型XML 使用Nokogiri创建XML节点 用 nokogiri 修改和包裹 xml 节点 使用Nokogiri读取大型XML文件 使用Heroku存储和处理大型XML文件？ Nokogiri :: XML :: Reader＃encoding给“ nil” 合并Nokogiri中的两个XML文件
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM