具有Apache Camel +聚合器的大型CSV

Question

生成與Apache駱駝與匯聚大文件具有更好的性能，因為這帖子： 與Apache駱駝大文件

我的身體有33352行..並且使用completionSize="1000"和completionTimeout="2500" ，最終文件錯過了最后352行

<camel:split streaming="true">
  <camel:simple>${body}</camel:simple>
  <camel:marshal>
    <camel:csv quote='"' quoteDisabled="false" headerDisabled="true" />
  </camel:marshal>
  <camel:aggregate strategyRef="setfepCsvStringBodyAggregator" completionSize="1000" completionTimeout="2500">
    <camel:correlationExpression>
      <constant>true</constant>
    </camel:correlationExpression>
    <to uri="file:{{setfep_dir_inprogress}}/?fileName={{setfep_filename_clientes}}.txt&amp;fileExist=Append" />
  </camel:aggregate>
</camel:split>

最終文件有33000行，缺少352。

[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000

如果我使用completionSize="1000"和completionInterval="2500"我的最終文件將缺少33155個方法，缺少197行。

[Camel thread #0 - AggregateTimeoutChecker] [INFO ]  CamelLogger.log - Complete by=interval rows=566
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[Camel thread #0 - AggregateTimeoutChecker] [INFO ]  CamelLogger.log - Complete by=interval rows=43
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[Camel thread #0 - AggregateTimeoutChecker] [INFO ]  CamelLogger.log - Complete by=interval rows=401
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[Camel thread #0 - AggregateTimeoutChecker] [INFO ]  CamelLogger.log - Complete by=interval rows=768
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[Camel thread #0 - AggregateTimeoutChecker] [INFO ]  CamelLogger.log - Complete by=interval rows=377
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000

如何解決這個問題？

Apache Camel 2.19.0

Answer 1

疑難雜症！

我運行的單元測試...的背景下完成之前結束completionInterval或completionTimeout

當我在路線上延遲時，我可以看到總共寫入了33352行。

<camel:delay>
  <constant>5000</constant>
</camel:delay>

但是在生產中不是必需的，上下文可以繼續運行，或者我們可以使用forceCompletionOnStop選項：

<camel:aggregate strategyRef="setfepCsvStringBodyAggregator" forceCompletionOnStop="true" completionSize="1000" completionInterval="4000">