Jsoup 在 Google App Engine 上运行不正确

Question

I try to login facebook with JSoup, when test with console app or Tomcat server webapp, it runs well:

String userAgent = BrowserVersion.FIREFOX_38.getUserAgent();
Connection conn1 = Jsoup.connect("https://m.facebook.com");
Document document = conn1.userAgent(userAgent).execute().parse();
Element form = document.select("form").get(0);
Elements inputs = form.select("input");
Connection conn2 = Jsoup.connect(form.absUrl("action")).userAgent(userAgent);
for (Element input: inputs) {
    conn2.data(input.attr("name"), input.val());
}
conn2.data("email", "mail@gmail.com").data("pass", "mypass");
Connection.Response response1 = conn2.method(Connection.Method.POST).execute();
System.out.println(response1.url());
Document doc1 = response1.parse();
Elements h3 = doc1.select("h3");
for (Element element: h3) {
    System.out.println(element.text());
}

response1.url() is https://m.facebook.com/home.php?_rdr

But when I try with Google App Engine app, it cannot login but shows the page with message "you have to login first" (I guess that the request try to access some other url)

response1.url() is https://m.facebook.com/login.php?next=https%3A%2F%2Fm.facebook.com%2Fhome.php%3Frefsrc%3Dhttps%253A%252F%252Fm.facebook.com%252F%26refid%3D8&refsrc=https%3A%2F%2Fm.facebook.com%2F&refid=8&_rdr

I change the password to make it incorrect, it shows the incorrect-password page. So I think with correct password, the request goes to the proper page. But why with correct password, it redirects to other page? I don't think there is any different thing between console app and GAE that makes the request different!

Answer 1

App Engine can do a few strange things from time to time when you're creating outbound HTTP requests from the server. Depending on how JSoup implements the underlying request, it's possible something is getting mixed up. Under the hood, it would likely need to use urlfetch .

One thing you could try here is to use App Engine Flexible . Outbound requests aren't going to be piped through UrlFetch or the Sockets API there, since ultimately your app is just running on Google Compute Engine.

If your app isn't using a ton of the App Engine APIs, this could be an easy way to get around the issue.

Hope this helps!

Answer 2

I would suggest that after successful login, facebook returns some redirect (several sequential redirects?) as a response.

When you run your app outside GAE, java just follows those redirects automatically, and eventually returns https://m.facebook.com/home.php?_rdr as a result.

On the other hand, when running inside GAE, GAE's URLFetch library follows up to 5 redirects only (or none, if configured so).

If that's the case, then you could follow those redirects in your code.

Can you also print response1.statusCode() to check this hypothesis?

Answer 3

因为这是处理某种身份验证，我想知道您是否需要保留第一个响应中的 cookie 并在后续请求中利用它们来提供该身份验证。