How to fix “java.nio.charset.MalformedInputException: Input length = 1” in Camel route?

Question

I have a Camel route which polls an ftp server for a zip file, extracts a csv file from the zip, then sends an http request for each row of the csv as a json body. Some csv files are <10MB in size and some are about 500MB. The problem is that I cannot process the smaller files fully, some rows are sent then I get MalformedInputException and a rollback starts.

So far the only way I have found to fix this is to convertBodyTo(String.class) before splitting. This fixes the issue with the smaller files however then the larger files cause a heap space error because the whole file gets loaded into memory.

    DataFormat dataFormat = BindyCsvDataFormat(myModel.class);

    from("ftp://" + username + "@" + ftpUrl + "?password=" + password + "&noop=True" +
            "&binary=True&sortBy=reverse:file:modified&antInclude=" + filePrefix + "*" +
            "&eagerMaxMessagesPerPoll=false&maxMessagesPerPoll=1")
            .streamCaching()
            .unmarshal().zipFile()
            .convertBodyTo(String.class) // works for small files with this line, but not for large
            .split(body().tokenize("\r\n", 1, true))
            .streaming()
            .unmarshal(dataFormat)
            .marshal().json(JsonLibrary.XStream)
            .to(outputUrl)
            .end();

Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
org.apache.camel.RuntimeCamelException: Scanner aborted because of an IOException!
    at org.apache.camel.processor.Splitter$SplitterIterable$1.hasNext(Splitter.java:175) ~[camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.processor.MulticastProcessor$MulticastState.run(MulticastProcessor.java:304) ~[camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.impl.engine.DefaultReactiveExecutor$3.run(DefaultReactiveExecutor.java:116) ~[camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:185) [camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:59) [camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:87) [camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:228) [camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:454) [camel-file-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:223) [camel-file-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:186) [camel-file-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.support.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:172) [camel-support-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.support.ScheduledPollConsumer.run(ScheduledPollConsumer.java:99) [camel-support-3.0.0-RC1.jar:3.0.0-RC1]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.nio.charset.MalformedInputException: Input length = 1
    at java.nio.charset.CoderResult.throwException(CoderResult.java:274) ~[?:?]
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339) ~[?:?]
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) ~[?:?]
    at java.io.InputStreamReader.read(InputStreamReader.java:185) ~[?:?]
    at java.io.Reader.read(Reader.java:189) ~[?:?]
    at org.apache.camel.util.Scanner.readMore(Scanner.java:162) ~[camel-util-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.util.Scanner.next(Scanner.java:136) ~[camel-util-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.util.Scanner.next(Scanner.java:44) ~[camel-util-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.support.GroupTokenIterator.doNext(GroupTokenIterator.java:122) ~[camel-support-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.support.GroupTokenIterator.next(GroupTokenIterator.java:112) ~[camel-support-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.processor.Splitter$SplitterIterable$1.next(Splitter.java:182) ~[camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.processor.Splitter$SplitterIterable$1.next(Splitter.java:158) ~[camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.processor.MulticastProcessor$MulticastState.run(MulticastProcessor.java:303) ~[camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    ... 16 more

java.nio.charset.MalformedInputException: Input length = 1
    at java.nio.charset.CoderResult.throwException(CoderResult.java:274) ~[?:?]
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339) ~[?:?]
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) ~[?:?]
    at java.io.InputStreamReader.read(InputStreamReader.java:185) ~[?:?]
    at java.io.Reader.read(Reader.java:189) ~[?:?]
    at org.apache.camel.util.Scanner.readMore(Scanner.java:162) ~[camel-util-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.util.Scanner.next(Scanner.java:136) ~[camel-util-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.util.Scanner.next(Scanner.java:44) ~[camel-util-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.support.GroupTokenIterator.doNext(GroupTokenIterator.java:122) ~[camel-support-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.support.GroupTokenIterator.next(GroupTokenIterator.java:112) ~[camel-support-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.processor.Splitter$SplitterIterable$1.next(Splitter.java:182) ~[camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.processor.Splitter$SplitterIterable$1.next(Splitter.java:158) ~[camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.processor.MulticastProcessor$MulticastState.run(MulticastProcessor.java:303) [camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.impl.engine.DefaultReactiveExecutor$3.run(DefaultReactiveExecutor.java:116) [camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:185) [camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:59) [camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:87) [camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:228) [camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:454) [camel-file-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:223) [camel-file-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:186) [camel-file-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.support.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:172) [camel-support-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.support.ScheduledPollConsumer.run(ScheduledPollConsumer.java:99) [camel-support-3.0.0-RC1.jar:3.0.0-RC1]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:834) [?:?]```

Answer 1

The error message points to an encoding issue. Have the two different types of files different encodings ?

If yes, you would have to determine somehow what encoding a file has. Or at least know the different types you get and assign the correct encoding to a file.

If you know the encoding, you can set it as Exchange property as suggested by @bedla.

So if the small files are for example ISO encoded, you could try

.setProperty(Exchange.CHARSET_NAME, constant("ISO-8859-1"))