登录https网站并仅使用核心Java API下载页面

Question

I want to login to a https website with username and password, go to one url in that website and download the page at the url (and maybe parse contents of that page). I want to do this using only core Java apis and not htmlunit, jsoup etc. I got the below code to learn how to do this, but it does not show me how to login to a website. Please tell me how I can login, maintain a session and then finally close the connection.

Source - http://www.mkyong.com/java/java-https-client-httpsurlconnection-example/

import java.net.MalformedURLException;
import java.net.URL;
import java.security.cert.Certificate;
import java.io.*;

import javax.net.ssl.HttpsURLConnection;
import javax.net.ssl.SSLPeerUnverifiedException;

public class HttpsClient{

   public static void main(String[] args)
   {
        new HttpsClient().testIt();
   }

   private void testIt(){

      String https_url = "https://www.google.com/";
      URL url;
      try {

         url = new URL(https_url);
         HttpsURLConnection con = (HttpsURLConnection)url.openConnection();

         //dumpl all cert info
         print_https_cert(con);

         //dump all the content
         print_content(con);

      } catch (MalformedURLException e) {
         e.printStackTrace();
      } catch (IOException e) {
         e.printStackTrace();
      }

   }

   private void print_https_cert(HttpsURLConnection con){

    if(con!=null){

      try {

    System.out.println("Response Code : " + con.getResponseCode());
    System.out.println("Cipher Suite : " + con.getCipherSuite());
    System.out.println("\n");

    Certificate[] certs = con.getServerCertificates();
    for(Certificate cert : certs){
       System.out.println("Cert Type : " + cert.getType());
       System.out.println("Cert Hash Code : " + cert.hashCode());
       System.out.println("Cert Public Key Algorithm : " 
                                    + cert.getPublicKey().getAlgorithm());
       System.out.println("Cert Public Key Format : " 
                                    + cert.getPublicKey().getFormat());
       System.out.println("\n");
    }

    } catch (SSLPeerUnverifiedException e) {
        e.printStackTrace();
    } catch (IOException e){
        e.printStackTrace();
    }

     }

   }

   private void print_content(HttpsURLConnection con){
    if(con!=null){

    try {

       System.out.println("****** Content of the URL ********");            
       BufferedReader br = 
        new BufferedReader(
            new InputStreamReader(con.getInputStream()));

       String input;

       while ((input = br.readLine()) != null){
          System.out.println(input);
       }
       br.close();

    } catch (IOException e) {
       e.printStackTrace();
    }

       }

   }

}

Answer 1

Every website manages logins differently. You will need to scout the website, find out how the session is maintained, and mimic the functions in such a way that the server can't tell that it is not a browser.

In general, a web server stores a secret hash in the cookie. Here is the process

1. Post a login and password to said url using HttpsURLConnection to send the form.
2. The server responds with a hash in a header that it wants stored in the cookie. Usually has session in the name.
3. Send requests back with the hash in the header in the correct value

All of the above can be done only using URL and HttpsURLConnection, but you will need to mimic a browser exactly to trick the server.

For scouting, I would recommend using a tool like fiddler . It captures all communication from the webserver and back, so that you can see exactly what is going on at the http level to mimic in your java code.

Here is an overview of fiddler . I have never looked at the logs. Fiddler has a sweet interface. The video is really boring, but it gives an overview of the interface. You want to look at the raw text view, and mimic that.

For your other question, owasp is a great resource for best practices. The reality is that there is a lot of insecure and bad code out there that does stuff that you would never expect. I have seen a server put the boolean value inside of a script tag to be stored as a javascript variable. You just have to carefully watch how the server changes the responses after you log in. For a popular website following best practices, they will use the above method.