JSOUP HTML parsing from URL
Question
I'm using JSOUP in Java to parse HTMLs like these two: This and this .
In the first case, I get the output.
And I have a problem with the connection:
doc = Jsoup.connect(url).get();
There are some URLs which can easily be parsed, and I've got the output, but there are URLs too which produces empty output like this:
Title: [].
I can't understand what the problem is if both URLs are the same. This is my code:
Document doc;
try {
doc = Jsoup.connect("http://ekonomika.sme.sk/c/8047766/s-velkymi-chybami-stavali-aj-budovu-centralnej-banky.html").get();
String title = doc.title();
System.out.println("title : " + title);
}
catch (IOException e) {
e.printStackTrace();
}
2 answers
solution1
0 2015-10-24 22:17:08
Take a look at what's in the head of the second url
Element h = doc.head();
System.out.println("head : " + h);
You'll see there are some meta refresh tags and an empty title:
<head>
<noscript>
<meta http-equiv="refresh" content="1;URL='/c/8047766/s-velkymi-chybami-stavali-aj-budovu-centralnej-banky.html?piano_d=1'">
</noscript>
<meta http-equiv="refresh" content="10;URL='/c/8047766/s-velkymi-chybami-stavali-aj-budovu-centralnej-banky.html?piano_t=1'">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title></title>
</head>
Which explains the empty title. You have to follow the redirect.
solution2
0 2015-10-25 12:06:40
Here is my code for parsing, with this URL I have no output. /* * To change this license header, choose License Headers in Project Properties. * To change this template file, choose Tools | Templates * and open the template in the editor. */ package commentparser;
import java.io.IOException;
import static java.lang.Boolean.FALSE;
import static java.lang.Boolean.TRUE;
import java.net.URL;
import static java.sql.JDBCType.NULL;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashSet;
import java.util.Set;
import static javafx.beans.binding.Bindings.length;
import static jdk.nashorn.internal.objects.ArrayBufferView.length;
import static oracle.jrockit.jfr.events.Bits.length;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
public class CommentParser {
public static void main(String[] args) {
Document doc;
try {
doc = Jsoup.connect("http://ekonomika.sme.sk/c/8047766/s-velkymi-chybami-stavali-aj-budovu-centralnej-banky.html").followRedirects(true).get();
String title = doc.title();
System.out.println("title : " + title);
//Link for discussions
if(doc.select("a[href^=/diskusie/reaction_show]").isEmpty() == FALSE){
Elements description = doc.select("a[href^=/diskusie/reaction_show]");
for (Element link : description) {
// get the value from href attribute
System.out.println("Diskusie: " + link.attr("href"));
}
}
//Author of article
if(doc.select("span[class^=autor]").isEmpty() == FALSE){
Elements description = doc.select("span[class^=autor]");
for (Element link : description) {
// get the value from href attribute
//System.out.println("\nlink : " + link.attr("b"));
System.out.println(link.text());
}
}
// get all links
Elements links = doc.select("a[href]");
for (Element link : links) {
// get the value from href attribute
System.out.println("\nlink : " + link.attr("href"));
System.out.println("text : " + link.text());
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.