堆栈内存溢出
  简体   繁体   English

具有Apache Camel +聚合器的大型CSV

[英]Large CSV with Apache Camel + Aggregator

 原文  2017-06-07 21:44:17  2   1    csv/ apache-camel

问题描述

Generate big files with Apache Camel with Aggregator with better performance as this post: Large Files with Apache Camel 生成与Apache骆驼与汇聚大文件具有更好的性能，因为这帖子： 与Apache骆驼大文件

My body has 33352 rows.. and using completionSize="1000" and completionTimeout="2500" the final file missed the last 352 rows 我的身体有33352行..并且使用completionSize="1000"和completionTimeout="2500" ，最终文件错过了最后352行 

<camel:split streaming="true">
  <camel:simple>${body}</camel:simple>
  <camel:marshal>
    <camel:csv quote='"' quoteDisabled="false" headerDisabled="true" />
  </camel:marshal>
  <camel:aggregate strategyRef="setfepCsvStringBodyAggregator" completionSize="1000" completionTimeout="2500">
    <camel:correlationExpression>
      <constant>true</constant>
    </camel:correlationExpression>
    <to uri="file:{{setfep_dir_inprogress}}/?fileName={{setfep_filename_clientes}}.txt&amp;fileExist=Append" />
  </camel:aggregate>
</camel:split>

Final file has 33000 rows missing 352. 最终文件有33000行，缺少352。 

[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000

if I use completionSize="1000" and completionInterval="2500" my final file has 33155 hows missing 197 rows. 如果我使用completionSize="1000"和completionInterval="2500"我的最终文件将缺少33155个方法，缺少197行。 

[Camel thread #0 - AggregateTimeoutChecker] [INFO ]  CamelLogger.log - Complete by=interval rows=566
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[Camel thread #0 - AggregateTimeoutChecker] [INFO ]  CamelLogger.log - Complete by=interval rows=43
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[Camel thread #0 - AggregateTimeoutChecker] [INFO ]  CamelLogger.log - Complete by=interval rows=401
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[Camel thread #0 - AggregateTimeoutChecker] [INFO ]  CamelLogger.log - Complete by=interval rows=768
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[Camel thread #0 - AggregateTimeoutChecker] [INFO ]  CamelLogger.log - Complete by=interval rows=377
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000

How to fix this? 如何解决这个问题？

Apache Camel 2.19.0

1 个解决方案

解决方案1
 0  2017-06-07 22:14:49

gotcha! 疑难杂症！

I running as unit test... the context ends before complete completionInterval or completionTimeout 我运行的单元测试...的背景下完成之前结束completionInterval或completionTimeout

when I put a delay in my route I could see the total 33352 rows wrote. 当我在路线上延迟时，我可以看到总共写入了33352行。 

<camel:delay>
  <constant>5000</constant>
</camel:delay>

But in production isn't necessary, the context continue alive or we can use the option forceCompletionOnStop : 但是在生产中不是必需的，上下文可以继续运行，或者我们可以使用forceCompletionOnStop选项： 

<camel:aggregate strategyRef="setfepCsvStringBodyAggregator" forceCompletionOnStop="true" completionSize="1000" completionInterval="4000">

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 使用apache Camel处理非常大的CSV文件的最佳方法是什么？ - What is the best approach to processing very large CSV files with apache Camel? 使用Apache Camel插入大型CSV文件时出现GC问题 - GC problems when inserting large CSV file with Apache Camel Apache camel + csv + header - Apache camel + csv + header 带标题的Apache Camel CSV - Apache Camel CSV with Header Apache骆驼-SQL到CSV - Apache camel - SQL to CSV 骆驼大CSV文件处理问题 - Camel large CSV file processing Issue 如何使用Apache Camel聚合CSV行？ - How to aggregate CSV lines with Apache Camel? Apache骆驼逐行拆分csv - Apache camel split csv line by line 使用Apache骆驼，SQL创建CSV文件 - Using apache camel, sql to create csv files 将Apache Camel CSV与拆分器模式相结合 - Combine Apache Camel CSV with the splitter pattern
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM