Sidekiq job doesn't seem to be parsing

Question

I'm trying to mimic what a previous developer did to parse an XML file in my Rails app and am stuck. From what I can tell, my job completes, but nothing is being posted as it should be, so I'm guessing my parsing file is incorrect (however, it works fine when testing with the raw file on my localhost). So, where am I going wrong here?

This is Sidekiq log output, just to confirm job is happening and not showing any errors in processing:

2016-05-25T13:51:04.499Z 8977 TID-oxs3s9lng ParseTestData JID-2a01971539c887cac3bf3374:1 INFO: start
2016-05-25T13:51:04.781Z 8977 TID-oxs3s9l3g GenerateNotifications JID-2a01971539c887cac3bf3374:2 INFO: start
2016-05-25T13:51:04.797Z 8977 TID-oxs3s9lng ParseTestData JID-2a01971539c887cac3bf3374:1 INFO: done: 0.297 sec
2016-05-25T13:51:04.824Z 8977 TID-oxs3s9l3g GenerateNotifications JID-2a01971539c887cac3bf3374:2 INFO: done: 0.043 sec

This is my Sidekiq job file, which iterates through the compressed files that get submitted through my API. The file in question that I'm working on is nmap_poodle_scan.xml :

class ParseTestData
  include Sidekiq::Worker

  # Order matters. Parse network hosts first to ensure we uniquely identify network hosts by their mac address.
  PARSERS = {
    "network_hosts.xml" => Parsers::NetworkHostParser,
    "nmap_tcp_service_scan.xml" => Parsers::TcpServiceScanParser,
    "nmap_shellshock_scan.xml" => Parsers::ShellshockScanParser,
    "hydra.out" => Parsers::HydraParser,
    "events.log" => Parsers::EventParser,
    "nmap_poodle_scan.xml" => Parsers::PoodleScanParser
  }

  def perform(test_id)
    test = Test.find(test_id)

    gzip = if Rails.env.development?
      Zlib::GzipReader.open(test.data.path)
    else
      file = Net::HTTP.get(URI.parse(test.data.url))
      Zlib::GzipReader.new(StringIO.new(file))
    end

    # Collect entries from tarball
    entries = {}
    tar_extract = Gem::Package::TarReader.new(gzip)
    tar_extract.rewind
    tar_extract.each do |entry|
      entries[File.basename(entry.full_name)] = entry.read
    end

    # Preserve parse order by using the parser hash to initiate parser executions.
    PARSERS.each_pair do |filename, parser|
      next unless entry = entries[filename]
      parser.run!(test, entry)
    end
  end
end

Which grabs nmap_poodle_scan.xml:

<host starttime="1464180941" endtime="1464180941"><status state="up" reason="arp-response" reason_ttl="0"/>
<address addr="10.10.10.1" addrtype="ipv4"/>
<address addr="4C:E6:76:3F:2F:77" addrtype="mac" vendor="Buffalo.inc"/>
<hostnames>
<hostname name="DD-WRT" type="PTR"/>
</hostnames>
Nmap scan report for DD-WRT (10.10.10.1)
<ports><extraports state="closed" count="996">
<extrareasons reason="resets" count="996"/>
</extraports>
<table key="CVE-2014-3566">
<elem key="title">SSL POODLE information leak</elem>
<elem key="state">VULNERABLE</elem>
<table key="ids">
<elem>OSVDB:113251</elem>
<elem>CVE:CVE-2014-3566</elem>
</table>
<table key="description">
<elem>    The SSL protocol 3.0, as used in OpenSSL through 1.0.1i and&#xa;    other products, uses nondeterministic CBC padding, which makes it easier&#xa;    for man-in-the-middle attackers to obtain cleartext data via a&#xa;    padding-oracle attack, aka the &quot;POODLE&quot; issue.</elem>
</table>
<table key="dates">
<table key="disclosure">
<elem key="year">2014</elem>
<elem key="month">10</elem>
<elem key="day">14</elem>
</table>
</table>
<elem key="disclosure">2014-10-14</elem>
<table key="check_results">
<elem>TLS_RSA_WITH_3DES_EDE_CBC_SHA</elem>
</table>
<table key="refs">
<elem>https://www.imperialviolet.org/2014/10/14/poodle.html</elem>
<elem>http://osvdb.org/113251</elem>
<elem>https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-3566</elem>
<elem>https://www.openssl.org/~bodo/ssl-poodle.pdf</elem>
</table>
</table>
</script></port>
</ports>
<times srtt="4665" rttvar="556" to="100000"/>
</host>

Which should submit to PoodleScanParser:

module Parsers
  class PoodleScanParser < NmapScanParser
    def self.run!(test, content)    
      super(test, content, "//host//ports[.//elem[@key='state'][contains(text(), 'VULNERABLE')]]") do |host, network_host_test|
        logger.info "Something cool"
        IssueFinder.match(cve_id: "CVE-2014-3566").each do |issue|
          Result.generate!(network_host_test.id, issue.id)
        end
      end
    end
  end
end

Which inherits from NmapScanParser. This file is parser is confirmed to work fine, so I know it's not the issue:

module Parsers
  class NmapScanParser

    def self.run!(test, content, xpath)
      document = Nokogiri::XML(content)
      document.remove_namespaces!

      document.xpath(xpath).each do |host|
        ip_address = host.at_xpath("address[@addrtype='ipv4']").at_xpath("@addr").value
        vendor = host.at_xpath("address[@addrtype='mac']").at_xpath("@vendor").value rescue "Unknown"
        hostname = host.at_xpath("hostnames/hostname").at_xpath("@name").value rescue "Hostname Unknown"
        os = host.at_xpath("os/osmatch").at_xpath("@name").value rescue "Unknown"
        os_vendor = host.at_xpath("os/osmatch/osclass").at_xpath("@vendor").value rescue "Unknown"

        network_host_test = NetworkHostTest.generate!(test, ip_address: ip_address, hostname: hostname, vendor: vendor, os: os, os_vendor: os_vendor)

        # If we didn't find a network host, that's because our network_hosts file didn't have this entry.
        next unless network_host_test

        yield(host, network_host_test)
      end
    end

  end
end

I've confirmed the parser works on my localhost with the same raw output as above using a plain ruby file, and running ruby poodle_parser.rb :

require 'nokogiri'

document = Nokogiri::XML(File.open("poodle_results.xml"))
document.remove_namespaces!

document.xpath("//host[.//elem/@key='state']").each do |host|
  ip_address = host.at_xpath("address[@addrtype='ipv4']").at_xpath("@addr").value
  result =  host.at_xpath("//ports//elem[@key='state']").content
  puts "#{ip_address} #{result}"
end

Which outputs what I would expect in terminal:

10.10.10.1 VULNERABLE

So, in the end, I expect a Result to be generated, but it's not. I'm not seeing any errors in the Rails log on my localhost, nor am I seeing anything indicating an error in the Sidekiq logs either!

---

I decided to add a logger.info line to my PoodleScanParser to see if the Parser is even running as it should be. Assuming I did this correctly, the Parser doesn't look like it's running.

Answer 1

Well, the answer has nothing to do with Sidekiq, instead it was the output, which Nokogiri was dying on. Turns out Nmap was adding a non-XML line at the beginning of the XML file "Starting Nmap 7.12". So, Nokogiri was simply dying there.

I guess moral of the story is to make sure your XML output is what you Nokogiri intends it to be!