Nokogiri not working for advanced XPath search

Question

Take a look at this page: http://finviz.com/quote.ashx?t=AAPL

The XPath

(//*[contains(concat(' ', @class, ' '), ' snapshot-table2 ')]/tbody/tr/td[text() = 'Index']/following::td)[1]/b

should spit out S&P 500 , and when I try it out in the JavaScript console, it does. However, when I try using Nokogiri with Ruby, it returns an empty array.

Here's the full code:

url = "http://finviz.com/quote.ashx?t=#{ticker}"
data = Nokogiri::HTML(open(url))
xpath = "(//*[contains(concat(' ', @class, ' '), ' snapshot-table2 ')]/tbody/tr/td[text() = '#{datapoint}']/following::td)[1]/b"
data.xpath(xpath)

data.xpath(xpath) returns []

Any ideas?

Answer 1

In the response of that page, I don't see a <tbody> element.

You mentioned your XPath works in your browser console, this is likely because the browser is injecting the tbody element for rendering purposes, see " Why do browsers insert tbody element into table elements? " for detail on why this happens.

Try your XPath again without specifying the <tbody> node:

data.xpath("(//*[contains(@class, 'snapshot-table2')]/tr/td[text() = 'Index']/following::td)[1]/b")

You could also use something like this to get the same XPath to work in both the browser and your Ruby code:

(//*[contains(@class, 'snapshot-table2')]//tr/td[text() = 'Index']/following::td)[1]/b