简体繁体 English

批处理系统如何处理许多对象

[英]How batch processing systems deal with a lot of objects

原文 2015-06-21 23:07:37 8 1 java/ garbage-collection/ spring-batch/ batch-processing

I have a question in mind which I'll try to explain it as well as I can. 我有一个问题要尽力解释。

Batch processing frameworks such as spring batch deal with a handful amount of objects. 批处理框架（例如spring batch）处理少量对象。

They should process the objects part by part so they don't encounter java heap space error. 他们应该对对象进行部分处理，以免遇到Java堆空间错误。

What will these kind of frameworks or systems do with garbage they produces? 这些框架或系统将如何处理它们产生的垃圾？

Do they call System.gc() sometimes or they handle it in different ways? 他们有时会调用System.gc()还是以不同的方式处理它？

1 个解决方案

Garbage collection will be done prior to throwing an OutOfMemoryError , so you won't run out of memory because consumption is outpacing cleanup. 垃圾回收将在抛出OutOfMemoryError之前完成，因此不会消耗内存，因为消耗超过了清理速度。

The question is too broad to cover what frameworks might or might not do, but in general, it's good practice to hold references to objects for a little scope as possible. 这个问题太笼统了，无法涵盖框架可能会做什么或可能不会做什么，但是总的来说，优良作法是在尽可能小的范围内保留对对象的引用。 That means adopting a "stream" approach to processing - where all operations on an object occur (eg read from input stream, process, write to output) before the next object is processed. 这意味着要采用“流”方法进行处理-在对下一个对象进行处理之前，对对象的所有操作都会发生（例如，从输入流中读取，处理，写入输出）。

This is in contrast to reading in all objects, processing all objects then writing all output - which requires large memory usage. 这与读取所有对象，处理所有对象然后写入所有输出形成对比，这需要占用大量内存。