HttpClient 4.x httpGet with cookies/session

Question

I'm trying to get a page from a given URL which need to be authenticated to work.

I've succeeded to get cookies with a HttpPost on the login url. I get cookies like that, so it seems ok:

Cookie: [version: 0][name: ASP.NET_SessionId][value:wqvdz0bvroatgi45ywpsto2q][domain:xxx][path:/][expiry: null] 

Cookie: [version: 0][name: lan][value: 2057][domain:xxx][path: /][expiry: Thu Dec 13 02:08:52 CET 2012]

Now, when i try to get a page from a known url, i finally only get the "login" page.

I'm converting the code from PHP that works well. I've put what i think is the equivalent each time.

    HttpClient httpClient = new DefaultHttpClient();
    httpClient.getParams().setParameter(CoreProtocolPNames.HTTP_CONTENT_CHARSET, CHARSET);

    //      curl_setopt($ch2, CURLOPT_COOKIESESSION, false);
    httpClient.getParams().setParameter(ClientPNames.COOKIE_POLICY, CookiePolicy.IGNORE_COOKIES);

    //      curl_setopt($ch2, CURLOPT_FOLLOWLOCATION, true);
    httpClient.getParams().setParameter(ClientPNames.HANDLE_REDIRECTS,true);
    //      curl_setopt($ch2, CURLOPT_MAXREDIRS, 10);
    httpClient.getParams().setParameter(ClientPNames.MAX_REDIRECTS, 10);
    //      curl_setopt($ch2, CURLOPT_USERAGENT, USER_AGENT);
    httpClient.getParams().setParameter(CoreProtocolPNames.USER_AGENT,USER_AGENT); 
    //      curl_setopt($ch2, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
    httpClient.getParams().setParameter(CoreProtocolPNames.PROTOCOL_VERSION, HttpVersion.HTTP_1_1);


    //      curl_setopt($ch2, CURLOPT_COOKIEFILE, COOKIE_FILE);
    Connexion con = new Connexion();
    CookieStore oldCookieStore = con.getCookieStore();
    CookieStore cookieStore = new BasicCookieStore();
    for (Cookie c : oldCookieStore.getCookies()) {
        cookieStore.addCookie(c);
    }

    HttpContext context = new BasicHttpContext();
    context.setAttribute(ClientContext.COOKIE_STORE, cookieStore);

    HttpGet httpGet = new HttpGet(URL);

    HttpResponse response = httpClient.execute(httpGet, context);

    // Examine the response status
    System.out.println(response.getStatusLine());

    // Get hold of the response entity
    HttpEntity entity = response.getEntity();

    ... ( treat entity )

I haven't found the equivalent of those lines, i think it's inherent:

 // curl_setopt($ch2, CURLOPT_BINARYTRANSFER, true); // curl_setopt($ch2, CURLOPT_RETURNTRANSFER, true); // curl_setopt($ch2, CURLOPT_AUTOREFERER, true) 

Here are the logs, where we can see 3 attempts that finally end on the login URL:

DEBUG [org.apache.http.impl.conn.BasicClientConnectionManager] Get connection for route {}->http://xxx
DEBUG [org.apache.http.impl.conn.DefaultClientConnectionOperator] Connecting to xxx:80
DEBUG [org.apache.http.client.protocol.RequestAddCookies] CookieSpec selected: ignoreCookies
DEBUG [org.apache.http.client.protocol.RequestAddCookies] Cookie [version: 0][name: ASP.NET_SessionId][value: wqvdz0bvroatgi45ywpsto2q][domain: xxx][path: /][expiry: null] match [xxx:80/GENERAL.aspx]
DEBUG [org.apache.http.client.protocol.RequestAddCookies] Cookie [version: 0][name: lan][value: 2057][domain: xxx][path: /][expiry: Thu Dec 13 02:08:52 CET 2012] match [xxx:80/GENERAL.aspx]
DEBUG [org.apache.http.client.protocol.RequestAuthCache] Auth cache not set in the context
DEBUG [org.apache.http.client.protocol.RequestTargetAuthentication] Target auth state: UNCHALLENGED
DEBUG [org.apache.http.client.protocol.RequestProxyAuthentication] Proxy auth state: UNCHALLENGED
DEBUG [org.apache.http.impl.client.DefaultHttpClient] Attempt 1 to execute request
DEBUG [org.apache.http.impl.conn.DefaultClientConnection] Sending request: GET /GENERAL.aspx?id=6414 HTTP/1.1
DEBUG [org.apache.http.headers] >> GET /GENERAL.aspx?id=6414 HTTP/1.1
DEBUG [org.apache.http.headers] >> Host: xxx
DEBUG [org.apache.http.headers] >> Connection: Keep-Alive
DEBUG [org.apache.http.headers] >> User-Agent: User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0
DEBUG [org.apache.http.impl.conn.DefaultClientConnection] Receiving response: HTTP/1.1 302 Found
DEBUG [org.apache.http.headers] << HTTP/1.1 302 Found
DEBUG [org.apache.http.headers] << Cache-Control: private
DEBUG [org.apache.http.headers] << Content-Length: 229
DEBUG [org.apache.http.headers] << Content-Type: text/html; charset=utf-8
DEBUG [org.apache.http.headers] << Location: /Default.aspx?ReturnUrl=%2fGENERAL.aspx%3id=6414
DEBUG [org.apache.http.headers] << Server: Microsoft-IIS/7.5
DEBUG [org.apache.http.headers] << X-AspNet-Version: 2.0.50727
DEBUG [org.apache.http.headers] << X-Powered-By: ASP.NET
DEBUG [org.apache.http.headers] << Date: Tue, 13 Nov 2012 01:08:51 GMT
DEBUG [org.apache.http.impl.client.DefaultHttpClient] Connection can be kept alive indefinitely
DEBUG [org.apache.http.impl.client.DefaultRedirectStrategy] Redirect requested to location '/Default.aspx?ReturnUrl=%2fGENERAL.aspx%3id=6414'
DEBUG [org.apache.http.impl.client.DefaultHttpClient] Redirecting to 'http://xxx/Default.aspx?ReturnUrl=%2fGENERAL.aspx%3id=6414' via {}->http://xxx
DEBUG [org.apache.http.client.protocol.RequestAddCookies] CookieSpec selected: ignoreCookies
DEBUG [org.apache.http.client.protocol.RequestAddCookies] Cookie [version: 0][name: ASP.NET_SessionId][value: wqvdz0bvroatgi45ywpsto2q][domain: xxx][path: /][expiry: null] match [xxx:80/Default.aspx]
DEBUG [org.apache.http.client.protocol.RequestAddCookies] Cookie [version: 0][name: lan][value: 2057][domain: xxx][path: /][expiry: Thu Dec 13 02:08:52 CET 2012] match [xxx:80/Default.aspx]
DEBUG [org.apache.http.client.protocol.RequestAuthCache] Auth cache not set in the context
DEBUG [org.apache.http.client.protocol.RequestTargetAuthentication] Target auth state: UNCHALLENGED
DEBUG [org.apache.http.client.protocol.RequestProxyAuthentication] Proxy auth state: UNCHALLENGED
DEBUG [org.apache.http.impl.client.DefaultHttpClient] Attempt 2 to execute request
DEBUG [org.apache.http.impl.conn.DefaultClientConnection] Sending request: GET /Default.aspx?ReturnUrl=%2fGENERAL.aspx%3id=6414 HTTP/1.1
DEBUG [org.apache.http.headers] >> GET /Default.aspx?ReturnUrl=%2fGENERAL.aspx%3id=6414 HTTP/1.1
DEBUG [org.apache.http.headers] >> Host: xxx
DEBUG [org.apache.http.headers] >> Connection: Keep-Alive
DEBUG [org.apache.http.headers] >> User-Agent: User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0
DEBUG [org.apache.http.impl.conn.DefaultClientConnection] Receiving response: HTTP/1.1 302 Found
DEBUG [org.apache.http.headers] << HTTP/1.1 302 Found
DEBUG [org.apache.http.headers] << Cache-Control: private
DEBUG [org.apache.http.headers] << Content-Length: 165
DEBUG [org.apache.http.headers] << Content-Type: text/html; charset=utf-8
DEBUG [org.apache.http.headers] << Location: /LOGIN.aspx
DEBUG [org.apache.http.headers] << Server: Microsoft-IIS/7.5
DEBUG [org.apache.http.headers] << X-AspNet-Version: 2.0.50727
DEBUG [org.apache.http.headers] << X-Powered-By: ASP.NET
DEBUG [org.apache.http.headers] << Date: Tue, 13 Nov 2012 01:08:52 GMT
DEBUG [org.apache.http.impl.client.DefaultHttpClient] Connection can be kept alive indefinitely
DEBUG [org.apache.http.impl.client.DefaultRedirectStrategy] Redirect requested to location '/LOGIN.aspx'
DEBUG [org.apache.http.impl.client.DefaultHttpClient] Redirecting to 'http://xxx/LOGIN.aspx' via {}->http://xxx
DEBUG [org.apache.http.client.protocol.RequestAddCookies] CookieSpec selected: ignoreCookies
DEBUG [org.apache.http.client.protocol.RequestAddCookies] Cookie [version: 0][name: ASP.NET_SessionId][value: wqvdz0bvroatgi45ywpsto2q][domain: xxx][path: /][expiry: null] match [xxx:80/LOGIN.aspx]
DEBUG [org.apache.http.client.protocol.RequestAddCookies] Cookie [version: 0][name: lan][value: 2057][domain: xxx][path: /][expiry: Thu Dec 13 02:08:52 CET 2012] match [xxx:80/LOGIN.aspx]
DEBUG [org.apache.http.client.protocol.RequestAuthCache] Auth cache not set in the context
DEBUG [org.apache.http.client.protocol.RequestTargetAuthentication] Target auth state: UNCHALLENGED
DEBUG [org.apache.http.client.protocol.RequestProxyAuthentication] Proxy auth state: UNCHALLENGED
DEBUG [org.apache.http.impl.client.DefaultHttpClient] Attempt 3 to execute request
DEBUG [org.apache.http.impl.conn.DefaultClientConnection] Sending request: GET /LOGIN.aspx HTTP/1.1
DEBUG [org.apache.http.headers] >> GET /LOGIN.aspx HTTP/1.1
DEBUG [org.apache.http.headers] >> Host: xxx
DEBUG [org.apache.http.headers] >> Connection: Keep-Alive
DEBUG [org.apache.http.headers] >> User-Agent: User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0
DEBUG [org.apache.http.impl.conn.DefaultClientConnection] Receiving response: HTTP/1.1 200 OK
DEBUG [org.apache.http.headers] << HTTP/1.1 200 OK
DEBUG [org.apache.http.headers] << Cache-Control: no-cache
DEBUG [org.apache.http.headers] << Pragma: no-cache
DEBUG [org.apache.http.headers] << Content-Length: 66322
DEBUG [org.apache.http.headers] << Content-Type: text/html; charset=utf-8
DEBUG [org.apache.http.headers] << Server: Microsoft-IIS/7.5
DEBUG [org.apache.http.headers] << X-AspNet-Version: 2.0.50727
DEBUG [org.apache.http.headers] << X-Powered-By: ASP.NET
DEBUG [org.apache.http.headers] << Date: Tue, 13 Nov 2012 01:08:52 GMT
DEBUG [org.apache.http.impl.client.DefaultHttpClient] Connection can be kept alive indefinitely
HTTP/1.1 200 OK
DEBUG [org.apache.http.impl.conn.BasicClientConnectionManager] Releasing connection org.apache.http.impl.conn.ManagedClientConnectionImpl@4cf7c31d
DEBUG [org.apache.http.impl.conn.BasicClientConnectionManager] Connection can be kept alive indefinitely

Answer 1

Your code:

   httpClient.getParams().setParameter(ClientPNames.COOKIE_POLICY, CookiePolicy.IGNORE_COOKIES);

is causing HttpClient to not store any cookies.

I assume that the server tracks your login via cookies, so since you're not storing any, you always appear to be unauthenticated, hence the login page.