如何在骆驼路线中解决“ java.nio.charset.MalformedInputException：输入长度= 1”？

Question

I have a Camel route which polls an ftp server for a zip file, extracts a csv file from the zip, then sends an http request for each row of the csv as a json body. 我有一个骆驼路线，该路线会轮询ftp服务器以获取zip文件，从zip中提取一个csv文件，然后将csv的每一行作为json正文发送一个http请求。 Some csv files are <10MB in size and some are about 500MB. 某些csv文件的大小小于10MB，而某些文件的大小约为500MB。 The problem is that I cannot process the smaller files fully, some rows are sent then I get MalformedInputException and a rollback starts. 问题是我无法完全处理较小的文件，发送了一些行，然后出现MalformedInputException并开始回滚。

So far the only way I have found to fix this is to convertBodyTo(String.class) before splitting. 到目前为止，我发现解决此问题的唯一方法是在拆分之前将convertBodyTo（String.class）修改 。 This fixes the issue with the smaller files however then the larger files cause a heap space error because the whole file gets loaded into memory. 这样可以解决较小文件的问题，但是较大文件会导致堆空间错误，因为整个文件都已加载到内存中。

    DataFormat dataFormat = BindyCsvDataFormat(myModel.class);

    from("ftp://" + username + "@" + ftpUrl + "?password=" + password + "&noop=True" +
            "&binary=True&sortBy=reverse:file:modified&antInclude=" + filePrefix + "*" +
            "&eagerMaxMessagesPerPoll=false&maxMessagesPerPoll=1")
            .streamCaching()
            .unmarshal().zipFile()
            .convertBodyTo(String.class) // works for small files with this line, but not for large
            .split(body().tokenize("\r\n", 1, true))
            .streaming()
            .unmarshal(dataFormat)
            .marshal().json(JsonLibrary.XStream)
            .to(outputUrl)
            .end();

Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
org.apache.camel.RuntimeCamelException: Scanner aborted because of an IOException!
    at org.apache.camel.processor.Splitter$SplitterIterable$1.hasNext(Splitter.java:175) ~[camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.processor.MulticastProcessor$MulticastState.run(MulticastProcessor.java:304) ~[camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.impl.engine.DefaultReactiveExecutor$3.run(DefaultReactiveExecutor.java:116) ~[camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:185) [camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:59) [camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:87) [camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:228) [camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:454) [camel-file-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:223) [camel-file-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:186) [camel-file-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.support.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:172) [camel-support-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.support.ScheduledPollConsumer.run(ScheduledPollConsumer.java:99) [camel-support-3.0.0-RC1.jar:3.0.0-RC1]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.nio.charset.MalformedInputException: Input length = 1
    at java.nio.charset.CoderResult.throwException(CoderResult.java:274) ~[?:?]
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339) ~[?:?]
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) ~[?:?]
    at java.io.InputStreamReader.read(InputStreamReader.java:185) ~[?:?]
    at java.io.Reader.read(Reader.java:189) ~[?:?]
    at org.apache.camel.util.Scanner.readMore(Scanner.java:162) ~[camel-util-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.util.Scanner.next(Scanner.java:136) ~[camel-util-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.util.Scanner.next(Scanner.java:44) ~[camel-util-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.support.GroupTokenIterator.doNext(GroupTokenIterator.java:122) ~[camel-support-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.support.GroupTokenIterator.next(GroupTokenIterator.java:112) ~[camel-support-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.processor.Splitter$SplitterIterable$1.next(Splitter.java:182) ~[camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.processor.Splitter$SplitterIterable$1.next(Splitter.java:158) ~[camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.processor.MulticastProcessor$MulticastState.run(MulticastProcessor.java:303) ~[camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    ... 16 more

java.nio.charset.MalformedInputException: Input length = 1
    at java.nio.charset.CoderResult.throwException(CoderResult.java:274) ~[?:?]
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339) ~[?:?]
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) ~[?:?]
    at java.io.InputStreamReader.read(InputStreamReader.java:185) ~[?:?]
    at java.io.Reader.read(Reader.java:189) ~[?:?]
    at org.apache.camel.util.Scanner.readMore(Scanner.java:162) ~[camel-util-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.util.Scanner.next(Scanner.java:136) ~[camel-util-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.util.Scanner.next(Scanner.java:44) ~[camel-util-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.support.GroupTokenIterator.doNext(GroupTokenIterator.java:122) ~[camel-support-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.support.GroupTokenIterator.next(GroupTokenIterator.java:112) ~[camel-support-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.processor.Splitter$SplitterIterable$1.next(Splitter.java:182) ~[camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.processor.Splitter$SplitterIterable$1.next(Splitter.java:158) ~[camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.processor.MulticastProcessor$MulticastState.run(MulticastProcessor.java:303) [camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.impl.engine.DefaultReactiveExecutor$3.run(DefaultReactiveExecutor.java:116) [camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:185) [camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:59) [camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:87) [camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:228) [camel-base-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:454) [camel-file-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:223) [camel-file-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:186) [camel-file-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.support.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:172) [camel-support-3.0.0-RC1.jar:3.0.0-RC1]
    at org.apache.camel.support.ScheduledPollConsumer.run(ScheduledPollConsumer.java:99) [camel-support-3.0.0-RC1.jar:3.0.0-RC1]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:834) [?:?]```

Answer 1

The error message points to an encoding issue. 该错误信息指出了编码问题。 Have the two different types of files different encodings ? 两种不同类型的文件的编码是否不同？

If yes, you would have to determine somehow what encoding a file has. 如果是，则必须以某种方式确定文件的编码方式。 Or at least know the different types you get and assign the correct encoding to a file. 或者至少知道您获得的不同类型，然后为文件分配正确的编码。

If you know the encoding, you can set it as Exchange property as suggested by @bedla. 如果您知道编码，则可以按照@bedla的建议将其设置为Exchange属性。

So if the small files are for example ISO encoded, you could try 因此，如果小文件是例如ISO编码的，则可以尝试

.setProperty(Exchange.CHARSET_NAME, constant("ISO-8859-1"))

如何在骆驼路线中解决“ java.nio.charset.MalformedInputException：输入长度= 1”？

问题描述

1 个解决方案

解决方案1
0 2019-09-09 15:37:07

如何在骆驼路线中解决“ java.nio.charset.MalformedInputException：输入长度= 1”？

问题描述

1 个解决方案

解决方案1 0 2019-09-09 15:37:07

解决方案1
0 2019-09-09 15:37:07