Encoding of Response is incorrect using Apache HttpClient

Question

I am calling a restful service that returns JSON using the Apache HttpClient.

The problem is I am getting different results in the encoding of the response when I run the code on different platforms.

Here is my code:

GetMethod get = new GetMethod("http://urltomyrestservice");
get.addRequestHeader("Content-Type", "text/html; charset=UTF-8");
...
HttpResponse response = httpexecutor.execute(request, conn, context);
response.setParams(params);
httpexecutor.postProcess(response, httpproc, context);
StringWriter writer = new StringWriter();
IOUtils.copy(response.getEntity().getContent(), writer);

When I run this on OSX, asian characters etc return fine eg 張惠妹 in the response. But when I run this on a linux server the same code displays the characters as ???

The linux server is an Amazon EC2 instance running Java 1.6.0_26-b03 My local OSX is running 1.6.0_29-b11

Any ideas really appreciated!!!!!

Answer 1

If you look at the javadoc of org.apache.commons.io.IOUtils.copy(InputStream, Writer) :

Copy bytes from an InputStream to chars on a Writer using the default character encoding of the platform.

So that will give different answers depending on the client (which is what you're seeing)

Also, Content-Type is usually a response header (unless you're using POST or PUT). The server is likely to ignore it (though you might have more luck with the Accept-Charset request header).

You need to parse the content type's charset-encoding parameter of the response header, and use that to convert the response into a String (if it's a String you're actually after). I expect Commons HTTP has code that will do that automatically for you. If it doesn't, Spring's RESTTemplate definitely does.

Answer 2

I believe that the problem is not in the HTTP encoding but elsewhere (eg while reading or forming the answer). Where do you get the content from and how? Is this stored in a DB or file?