HttpURLConnection is not returning all headers

Question

I'm printing HTTP headers using the following code.

    URL url = new Url("htttp://example.com/example");
    HttpURLConnection conn = (HttpURLConnection) url.openConnection();

    Map<String, List<String>> map = conn.getHeaderFields();
    for (Map.Entry<String, List<String>> entry : map.entrySet()) {
        System.out.println("Key : " + entry.getKey() +
                " ,Value : " + entry.getValue());
    }

Here's the output :

Key : null ,Value : [HTTP/1.1 200 OK]
Key : Accept-Ranges ,Value : [bytes]
Key : Cache-Control ,Value : [max-age=604800, public]
Key : Connection ,Value : [Keep-Alive]
Key : Date ,Value : [Mon, 03 Oct 2016 18:01:06 GMT]
Key : ETag ,Value : ["159eb4-53dce1f957880-gzip"]
Key : Expires ,Value : [Mon, 10 Oct 2016 18:01:06 GMT]
Key : Keep-Alive ,Value : [timeout=5, max=100]
Key : Last-Modified ,Value : [Sat, 01 Oct 2016 13:59:46 GMT]
Key : Server ,Value : [Apache/2.4.12 (Unix) OpenSSL/1.0.1e-fips mod_bwlimited/1.4]
Key : Transfer-Encoding ,Value : [chunked]
Key : Vary ,Value : [Accept-Encoding,User-Agent]

Now I'm trying via curl:

$ curl -v -D - http://example.com/example -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 111.111.111.111...
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0* Connected to example.com (111.111.111.111) port 80 (#0)
> GET example.com/example HTTP/1.1
> Host: example.com
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Mon, 03 Oct 2016 18:25:11 GMT
Date: Mon, 03 Oct 2016 18:25:11 GMT
< Server: Apache/2.4.12 (Unix) OpenSSL/1.0.1e-fips mod_bwlimited/1.4
Server: Apache/2.4.12 (Unix) OpenSSL/1.0.1e-fips mod_bwlimited/1.4
< Last-Modified: Sat, 01 Oct 2016 13:59:46 GMT
Last-Modified: Sat, 01 Oct 2016 13:59:46 GMT
< ETag: "159eb4-53dce1f957880"
ETag: "159eb4-53dce1f957880"
< Accept-Ranges: bytes
Accept-Ranges: bytes
< Content-Length: 1416884
Content-Length: 1416884
< Cache-Control: max-age=604800, public
Cache-Control: max-age=604800, public
< Expires: Mon, 10 Oct 2016 18:25:11 GMT
Expires: Mon, 10 Oct 2016 18:25:11 GMT
< Vary: Accept-Encoding,User-Agent
Vary: Accept-Encoding,User-Agent

< 
{ [1080 bytes data]
 30 1383k   30  427k    0     0   116k      0  0:00:11  0:00:03  0:00:08  116k^C

See how the Content-Length header is missing in the java output? Why does this happen? How can I fix this?

Answer 1

Content-Length is missing because the server sent the response in chunks. It's the expected behavior. Look at the Transfer-Encoding header in the Java response, and here's RFC 2616, 4.4 Message Length (see the 3rd item of the list).

Answer 2

The 'Content-Length' is parsed automatically and can be retrieved via conn.getContentLength() or conn.getContentLengthLong() . See https://docs.oracle.com/javase/7/docs/api/java/net/URLConnection.html#getContentLengthLong() for more info.

Answer 3

The response in case of java includes 'transfer-encoding' header which is missing in case of the curl response. Also when I tried it, I got a 404 (not found) for my attempt. I get a 404 when I try it from my browser as well.

My guess is that you have an overridden host entry for example.com that loops back to your own tomcat or similar. Its an issue with your environment.

This is what I get :

    /**
     *
     * <P> jdk-1.8 </P>
     * <PRE>
Key :null || Values :[HTTP/1.1 404 Not Found]
Key :X-Cache || Values :[HIT]
Key :Server || Values :[ECS (ewr/1445)]
Key :Etag || Values :["359670651+gzip+ident"]
Key :Cache-Control || Values :[max-age=604800]
Key :x-ec-custom-error || Values :[1]
Key :Vary || Values :[Accept-Encoding]
Key :Last-Modified || Values :[Fri, 09 Aug 2013 23:54:35 GMT]
Key :Expires || Values :[Mon, 10 Oct 2016 19:20:29 GMT]
Key :Content-Length || Values :[1270]
Key :Date || Values :[Mon, 03 Oct 2016 19:20:29 GMT]
Key :Content-Type || Values :[text/html]
     * </PRE>
     * @param urlStr (http://example.com/example)
     */
    private static void printHeaders(String urlStr) {

        try {
            URL urlRef = new URL(urlStr);
            URLConnection urlConnection = urlRef.openConnection();
            Map<String, List<String>> headerFields = urlConnection.getHeaderFields();

            for (String headerName : headerFields.keySet()) {
                List<String> headerEntry = headerFields.get(headerName);
                System.out.println("Key :"+headerName +" || Values :"+headerEntry);
            }
        } catch (MalformedURLException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }

    }