[英]Jsoup meta refresh redirect
I want to get an HTML page from a meta refresh redirect very similar as in question can jsoup handle meta refresh redirect . 我想从元刷新重定向获取HTML页面,就像jsoup可以处理元刷新重定向一样 。
But I can't get it to work. 但是我无法正常工作。 I want to do a search on http://synchronkartei.de .
我想在http://synchronkartei.de上进行搜索。 I have the following code:
我有以下代码:
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
public class SynchronkarteiScraper {
public static void main(String[] args) throws Exception{
Document doc = Jsoup.connect("https://www.synchronkartei.de/search.php")
.data("cat", "2")
.data("search", "Thomas Danneberg")
.data("action", "search")
.followRedirects(true)
.get();
Elements meta = doc.select("html head meta");
for (final Element m : meta){
if (m.attr("http-equiv").contains("refresh")){
doc = Jsoup.connect(m.baseUri()+m.attr("content").split("=")[1]).get();
}
}
System.out.println(doc.body().toString());
}
}
This does the search, which leads to a temporary site that gets refreshed opens the real result page. 进行搜索,从而导致刷新的临时站点打开了实际结果页面。 It is the same as going to http://synchronkartei.de , selecting "Sprecher" from the dropdownbox, entering "Thomas Danneberg" to the textfield and hitting enter.
与转到http://synchronkartei.de相同,从下拉框中选择“ Sprecher”,在文本字段中输入“ Thomas Danneberg”,然后按Enter。
But even after extracting the refresh URL and do a second connect, I still get the content of the temporary landing page, which can be seen in the prinln of the body. 但是即使提取了刷新URL并进行了第二次连接,我仍然可以获得临时登录页面的内容,该内容可以在正文中看到。
So what is going wrong here? 那么,这里出了什么问题?
As a note, the site synchronkartei.de always redirects to HTTPS. 请注意,站点syncnkartei.de始终重定向到HTTPS。 And since it is using a certificate from StartCom, java complains about the certificate path.
由于它使用的是StartCom的证书,因此Java抱怨证书路径。 To let the above code snippet work, it is necessary to use the VM parameter
-Djavax.net.ssl.trustStore=<path-to-keystore>
with the correct certificate. 要使上述代码段起作用,必须将VM参数
-Djavax.net.ssl.trustStore=<path-to-keystore>
与正确的证书一起使用。
I have to admit, that I am no expert in Jsoup, but I know some details about the Synchronkartei, though. 我不得不承认,我不是Jsoup的专家,但是我知道有关Synchronkartei的一些细节。
Deutsche Synchronkartei supports OpenSearchDescriptions, which is linked at /search.xml. Deutsche Synchronkartei支持OpenSearchDescriptions,该链接链接到/search.xml。 That said, you could also use
https://www.synchronkartei.de/search.php?search={searchTerms}
to get your search term into the session. 也就是说,您还可以使用
https://www.synchronkartei.de/search.php?search={searchTerms}
将搜索字词添加到会话中。
All you need is a cookie "sid" with the session ID, the Synchronkartei provides you. 您只需要一个带有会话ID的cookie“ sid”,Synchronkartei即可为您提供。 After that, a direct request to
https://www.synchronkartei.de/index.php?action=search
will provide you the results, regardless of your referrer. 之后,直接发送至
https://www.synchronkartei.de/index.php?action=search
请求将为您提供结果,而不管您的推荐人如何。
What I mean is, first send a request to https://www.synchronkartei.de/search.php?search={searchTerms}
or https://www.synchronkartei.de/search.php?cat={Category}&search={searchTerms}&action=search
(as you did above) and ignore the result completely if it has an HTTP result of 200, but safe the session cookie. 我的意思是,首先将请求发送到
https://www.synchronkartei.de/search.php?search={searchTerms}
或https://www.synchronkartei.de/search.php?cat={Category}&search={searchTerms}&action=search
(如上所述),如果HTTP结果为200,但完全忽略了会话cookie,则完全忽略该结果。 After that, you place a request to https://www.synchronkartei.de/index.php?action=search
which should provide you the whole list of results then. 之后,您向
https://www.synchronkartei.de/index.php?action=search
发出请求,然后该请求将为您提供整个结果列表。
Funzi 丰子
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.