用Java和jsoup进行网站爬取； html无法读取； 空指针异常

Question

I am trying to scrap data from a website using Java and jsoup. The main aim of my program is to read data out of a table. Unfortunately, the code works for a simple example table like this . But not for others like the one in the code.

 import org.jsoup.*;
 import org.jsoup.helper.*;
 import org.jsoup.nodes.*;
 import org.jsoup.select.*;
 import java.io.*; // Only needed if scraping a local File.
 import java.util.*;

 public class Test1 {
    public static void main(String args[]) throws IOException { 
        try{

            Document doc = Jsoup.connect("http://www.truckit.net/freight/details/index/id/62674").timeout(10*1000).get();
            String title = doc.title();

            Element table = doc.getElementById("table");
            Elements rows = table.getElementsByTag("tr");

            for (Element row : rows) {
                Elements tds = row.getElementsByTag("td");
                for (int i = 0; i < tds.size(); i++) {
                    if (i == 1) System.out.println(tds.get(i).text());
                }
            }                           
        }
        catch (java.io.IOException ex) {
            System.out.println("IO Error: " + ex);
            }
    }       
}

The console output is as follows:

Exception in thread "main" java.lang.NullPointerException
at Test1.main(Test1.java:30)

I read a number of threads about nullpointer exceptions but it did not really help me. I know that the variable table = null and thus the variable tr too, but why is that? As my program works for other websites, may my problem have to do with the websites html-code?

Answer 1

This is because the page in the link does not have an element which has the attribute id set to "table"

Meaning you'll have to create a different hook for the JSoup to latch onto data.

Tables will rarely have the id="table" attribute set since its redundant.

Thus generally you are better off with using

Elements tables = doc.getElementsByTag("table");

instead of:

Element table = doc.getElementById("table");

Especially since the page might have multiple tables available (as is the case in the website you mentioned)

Also note that scraping is a case-by-case kind of deal which means that each scraper will have to be tailor-made to a particular website or page, meaning that there is no one-size-fits-all scraper which will work everywhere.

Before attempting to scrape data you should examine the structure of the page you will want to scrape (through the view page source option) and then decide which data you'll want to scrape and what's the easiest pathway through the DOM to get it.