为什么chrome devtools中的html代码和jsoup解析的html代码不一样？

Question

I'm trying to extract information about created date of issues from HADOOP Jira issue site( https://issues.apache.org/jira/projects/HADOOP/issues/HADOOP-16381?filter=allopenissues )

As you can see in this Screenshot , created date is the text between the time tag whose class is live stamp(eg <time class=livestamp ...> 'this text' </time> )

So, I tried parse it with code as below.

import java.io.IOException;

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

public class CreatedDateExtractor {
    public static void main(String[] args) {
        String url = "https://issues.apache.org/jira/projects/HADOOP/issues/HADOOP-16381?filter=allopenissues";
        Document doc = null;

        try {
            doc = Jsoup.connect(url).get();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

        Elements elements = doc.select("time.livestamp"); //This line finds elements that matches time tags with livestamp class
        System.out.println("# of elements : "+ elements.size());
        for(Element e: elements) {
            System.out.println(e.text());
        }   
    }
}

I expect that created date is extracted, but the actual output is # of elements : 0 .

I found this is something wrong. So, I tried to parse whole html code from that side with below code.

import java.io.IOException;

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

public class CreatedDateExtractor {
    public static void main(String[] args) {
        String url = "https://issues.apache.org/jira/projects/HADOOP/issues/HADOOP-16381?filter=allopenissues";
        Document doc = null;

        try {
            doc = Jsoup.connect(url).get();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

        Elements elements = doc.select("*"); //This line finds whole elements in html document.
        System.out.println("# of elements : "+ elements.size());
        for(Element e: elements) {
            System.out.println(e);
        }   
    }
}

I compared both the html code in chrome devtools and the html code that I parsed one by one. Then I found those are different.

Can you explain why this happens and give me some advices how to extract created date?

Answer 1

I advice you to get elements with "time" tag, and use select to get time tags which have "livestamp" class. Here is the example:

Elements timeTags = doc.select("time");
Element timeLivestamp = null;
for(Element tag:timeTags){
  Element livestamp = tag.selectFirst(".livestamp");
  if(livestamp != null){
   timeLivestamp = livestamp;
   break;
   }

}

I don't know why but when I want to use .select() method of Jsoup with more than 1 selector (as you used like time.livestamp), I get interesting outputs like this.

Answer 2

import Torello.HTML.*;
import Torello.HTML.NodeSearch.*;
import Torello.Java.*;

import java.util.*;
import java.io.*;
import java.net.*;

public class Scrape
{
    public static void main(String[] argv) throws IOException
    {
        // This URL does not appear to have an HTML Element with a "TimeStamp" as you have stated.
        // ==> Go to any browser and view it for yourself!  (Click "View Source" in Google-Chrome, I.E., Safari, etc...)
        // URL url = new URL("https://issues.apache.org/jira/projects/HADOOP/issues/HADOOP-16381?filter=allopenissues");

        URL url = new URL("https://some.url.org/");

        // This scrapes the web-page into a standard Java-Vector.
        // HTMLNode is abstract, and has only 2 classes that inherit it.  (3 actually, but one is the "CommentNode")
        Vector<HTMLNode> page = HTMLPage.getPageTokens(url, false);

        // This will output each & every node in the page to a text/html file called "output.html"
        // Read Documentation Files for "Util.pageToString" and "FileRW.writeFile"
        FileRW.writeFile(Util.pageToString(page), "output.html");

        // If this is the question to identify:
        // As you can see in this Screenshot, created date is the text between the time tag whose class is
        // live stamp(e.g. <time class=livestamp ...> 'this text' </time>)
        //
        // Using the "NodeSearch.InnerTagGetInclusive" class will retrieve the information you need
        Vector<HTMLNode> liveStamp = InnerTagGetInclusive.first(page, "time", "class", TextComparitor.CN_CI, "livestamp");

        // This will get eliminate of all the "TagNode" elements when building a this String.
        // It will leave you with only the "TextNode" elements.
        // This remaining TextNode's should, indeed, be the the "this text" as a string.
        String liveStampStr = Util.textNodesString(liveStamp);

        System.out.println("Live-Stamp: " + liveStampStr);
    }
}