Getting incomplete json as result of toD(url) for few cases

Question

I am using apache camel and fetching data from data lake which is owned by another system. While in most of the cases this flow works absolutely fine but for very few events i have observed that incomplete data is received by my application and flow breaks saying com.fasterxml.jackson.databind.JsonMappingException: Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string value

I have check with the team who source the data - but they had already confirmed that no changes has happened at there end, which left me clueless how can i troubleshoot this issue further

if i use curl command and download the data from data lake i get the complete json is there any other way in apache camel which i can try to download the data from the dataSetAccessUrl? quick help will be appreciated as this is a prod issue

using: camel-spring-boot-bom > 3.9.0 camel-spring-boot-dependencies > 3.9.0

    from("direct:fetchData")
    .log(LoggingLevel.INFO, LOGGER, "Invoking API ")
    .toD("${header.dataSetAccessUrl}") // accessing CCO  data sets
    .log(LoggingLevel.INFO, LOGGER, "response code is: ${header.CamelHttpResponseCode}, JSON Response ${body}")
    .log(LoggingLevel.DEBUG, LOGGER, "JSON Response ${body}").end();

Answer 1

The com.fasterxml.jackson.databind.JsonMappingException: Illegal unquoted character ((CTRL-CHAR, code 0)): means that the JSON-response has illegal character. When this happens you should write the failed JSON response to a file and view it in VSCode or some other editor that supports syntax-highlighting for JSON.

from("direct:fetchData")
    .onException(JsonMappingException.class)
        .log(LoggingLevel.ERROR, "Json mapping failed. ${exception.message}")
        .to("file:{{fetchdata.mapping.failed}}")
    .end()
    // ...
;

Once had this happen with client when users had copy pasted data directly from pdf files which had control characters like STX (Start of text).

This is best fixed in the source to avoid having to clean this in every application using the REST-API, but you can also remove specific control characters from message body using replaceAll and regex.

// Removes stx control character from message body
String body = exchange.getMessage().getBody(String.class);
exchange.getMessage().setBody(body.replaceAll("[\\x02]", ""));

The .toD("header.dataSetAccessUrl") has nothing to do with this but you might want to validate that header at the start of the route. Like throwing exception when the header is null.