Ruby Nokogiri HTML抓取表与CSS问题

Question

I have an issue with the scraping of an html-table. Here is the link : https://www.basketball-reference.com/players/c/curryst01/gamelog/2016 (yes, it's a famous introductive tutorial for Ruby-scraping). Here is the code related :

doc = Nokogiri::HTML.parse(open(link))

# Get the biggest table 
big_table = doc.css("table").sort { |x,y| y.css("tr").count <=> x.css("tr").count }.first

# Number of rows is 87, but there are 5 heads that I wanna remove   
big_table.css("tr").count

# This doesn't remove heads 
big_table = big_table.select { |row| row.css("th").empty? }

In fact in HTML (I know nothing about HTML and i am in Ruby since 4h) th is the tag for header, td is for a standard cell, and tr is just a line. The goal was to delete the header, so as the .empty return if a nodeset (nodeset is like the content of a tag ? ) is empty, this last line of code should have return only the tr elements. But it doesn't work, in fact the result is [] .
Instead, I noticed that : big_table.select{|row| row.css("td").empty?}.count big_table.select{|row| row.css("td").empty?}.count was equal to 5 ... So, i decided to do :

big_table = big_table.select{|row| row.css("td").any?} big_table = big_table.select{|row| row.css("td").any?} and it worked well...

My question is : why did this line works ? and why the first attempt did fail ? Maybe it's something in the HTML structure that i'm missing ...

Thanks !

Answer 1

Let's take a look at big_table

> big_table.class
 => Nokogiri::XML::NodeSet

> big_table.size
 => 1

So first of all, doing Enumerable#select against big_table is probably not doing what you expect. If instead you capture the rows:

> rows = big_table.css("tr")
> rows.count
 => 87

Now you can do your select on the rows. Let's take an arbitrary row and see what it contains:

> rows[2].css("td").count
 => 29

> rows[2].css("th").count
 => 1

So a typical row has 29 td elements and one th . In fact every row has at least one th , which is why the css("th").empty? returned nothing. Conversely, the all-header rows do not contain any td elements, which is why what you tried worked.