HttpMessageNotReadableException: JSON parse error: Unrecognized token '嬀崀'

Question

I am calling an endpoint via RestTemplate as follows:

   SSLContext sslContext = SSLContexts.createSystemDefault();
   SSLConnectionSocketFactory socketFactory = new SSLConnectionSocketFactory(sslContext);
   HttpClient client = HttpClientBuilder.create().setSSLSocketFactory(socketFactory).build();
   ClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory(client);

   HttpEntity<String> entity = new HttpEntity<>(requestJson, headers());
   Object[] response = restTemplate.postForObject(uri, entity, Object[].class);

I have verified that the JSON String in the entity object is valid by copying it and using it in a cURL request to the same endpoint without any error. The same headers and authorization token were used in this request too.

When I execute the POST, I get back the following error:

Error while extracting response for type [class [Ljava.lang.Object;] and content type [application/json;charset=utf-16]

org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: Unrecognized token '嬀崀': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false'); nested exception is com.fasterxml.jackson.core.JsonParseException: Unrecognized token '嬀崀': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
     at [Source: (InputStreamReader); line: 1, column: 3]

My accept and content-type headers are both set to application/json . From examining the output from cURL, I see that there are no Chinese characters in the response body.

The response headers are as follows:

Connection →keep-alive
Content-Encoding →gzip
Content-Type →application/json; charset=utf-8
Date → xxx
Transfer-Encoding →chunked
Vary →Accept-Encoding

When I make the request with the responseType set to String.class or Object.class , the response is Chinese characters:

restTemplate.postForObject(uri, entity, String.class);
嬀崀

I am expecting that this call should return an empty array [] .

When I change the requestJson String to one which should serve back a non-empty array, I get back hundreds of Chinese characters instead of just two.

How can I decode the response to get valid data like when I use cURL?

edit:

I'm not sure how this is related, but the bytecode for the chars in an empty array [] are 91 and 93, and the bytecode for the two Chinese characters is 0, 91, 0, 93.

Answer 1

It seems like the response from restTemplate is returned as a String of Chinese characters not an array. The first error you posted seems to indicate that the issue is with extracting the response into an Object[] . If the response from the restTemplate is actually a String, then that would explain the second error as well. The RestTemplate was expecting to parse an array but instead received the String嬀崀. That's why changing the response type to String.class seems to work.

If you're expecting a JSON array back from the api you're calling, then I would double check the response from the api you're calling. Otherwise, I would suggest using String.class instead.

Edited: It's possible that the restTemplate is parsing the response using the utf-16 charset and the server is encoding the response using the utf-8 charset. Like you've posted in the description, those characters seem to have the same bytecode. Maybe changing the expected charset in the restTemplate to utf-8 will resolve your problem.

Answer 2

Do not use UTF16. HTTP Spec says ASCII, and many use UTF8. The error points this out with charset=utf-16 in it. Try setting the encoding header on the request.

What you're seeing, as you noted with the char codes, is exactly the result expected when using UTF16 because each char is 2 bytes.

Answer 3

The API is serving back UTF-8 content when I call it from cURL, but when I call the API programmatically it is serving back UTF-16LE despite my request headers asking for UTF-8 explicitly.

I will need to do some de-/en-coding games, but the following allows me to observe the valid, expected response JSON:

ResponseEntity<byte[]> responseEntity = restTemplate.postForEntity(uri, entity, byte[].class);
   
byte[] bytes = responseEntity.getBody();

String json = new String(bytes, StandardCharsets.UTF_16LE);

FooBar[] fooBar = objectMapper.readValue(json, FooBar[].class);