[英]Out of Memory exception while running multithreaded code
我正在開展一個項目,我將在其中使用不同的Bundles。 讓我們舉個例子,假設我有5個Bundle,每個bundle都有一個方法名稱process
。
現在,我正在使用下面的多線程代碼並行調用所有這5個bundle的process
方法。
但不知何故,每當我運行下面的多線程代碼時,它總是會讓我出現內存異常。 但是,如果我按順序運行它意味着,逐個調用進程方法,那么它不會給我任何Out Of memory異常。
以下是代碼 -
public void callBundles(final Map<String, Object> eventData) {
// Three threads: one thread for the database writer, two threads for the plugin processors
final ExecutorService executor = Executors.newFixedThreadPool(3);
final Map<String, String> outputs = (Map<String, String>)eventData.get(Constants.EVENT_HOLDER);
for (final BundleRegistration.BundlesHolderEntry entry : BundleRegistration.getInstance()) {
executor.submit(new Runnable () {
public void run() {
try {
final Map<String, String> response = entry.getPlugin().process(outputs);
//process the response and update database.
System.out.println(response);
} catch (Exception e) {
e.printStackTrace();
}
}
});
}
}
以下是例外情況,每當我在多線程代碼上運行時,我都會得到。
JVMDUMP006I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" - please wait.
JVMDUMP032I JVM requested Heap dump using 'S:\GitViews\Stream\goldseye\heapdump.20130904.175256.12608.0001.phd' in response to an event
JVMDUMP010I Heap dump written to S:\GitViews\Stream\goldseye\heapdump.20130904.175256.12608.0001.phd
JVMDUMP032I JVM requested Java dump using 'S:\GitViews\Stream\goldseye\javacore.20130904.175256.12608.0002.txt' in response to an event
UTE430: can't allocate buffer
UTE437: Unable to load formatStrings for j9mm
JVMDUMP010I Java dump written to S:\GitViews\Stream\goldseye\javacore.20130904.175256.12608.0002.txt
JVMDUMP032I JVM requested Snap dump using 'S:\GitViews\Stream\goldseye\Snap.20130904.175256.12608.0003.trc' in response to an event
UTE001: Error starting trace thread for "Snap Dump Thread": -1
JVMDUMP010I Snap dump written to S:\GitViews\Stream\goldseye\Snap.20130904.175256.12608.0003.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
ERROR: Bundle BullseyeModellingFramework [1] EventDispatcher: Error during dispatch. (java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 12)
java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 12
JVMDUMP006I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" - please wait.
JVMDUMP032I JVM requested Heap dump using 'S:\GitViews\Stream\goldseye\heapdump.20130904.175302.12608.0004.phd' in response to an event
JVMDUMP010I Heap dump written to S:\GitViews\Stream\goldseye\heapdump.20130904.175302.12608.0004.phd
JVMDUMP032I JVM requested Java dump using 'S:\GitViews\Stream\goldseye\javacore.20130904.175302.12608.0005.txt' in response to an event
我在我的eclipse中使用JDK1.6.0_26
作為已安裝的JRE。
每次調用callBundles()
都會通過創建一個自己的執行程序來創建一個新的callBundles()
。 每個線程都有自己的堆棧空間! 因此,如果你說你啟動JVM,第一次調用將創建三個線程,總和為3M堆(1024k是64位JVM的默認堆棧大小),下一次調用另一個3M等1000個調用/ s將需要3GB / S!
第二個問題是你永遠不會shutdown()
創建的執行器服務,所以線程將繼續運行,直到垃圾收集器刪除執行程序( finalize()
也調用shutdown()
)。 但是GC永遠不會清除堆棧內存,所以如果堆棧內存是問題並且堆未滿,GC將永遠無法幫助!
你需要使用一個 ExecutorService
,比方說10到30個線程或一個帶有3-30個緩存線程和LinkedBlockingQueue
的自定義ThreadPoolExecutor
。 如果可能,請在應用程序停止之前調用服務上的shutdown()
。
檢查應用程序的物理RAM,加載和響應時間,以調整參數堆大小,最大線程數並保持池中線程的活動時間。 查看代碼的其他鎖定部分(數據庫連接池的大小,......)以及服務器的CPU /核心數。 線程池大小的起始點可能是CPU數/核數加1.,更多I / O等待更有用。
主要問題是你沒有真正正確地使用線程池。 如果所有“進程”線程具有相同的優先級,則沒有充分的理由不創建一個大型線程池並將所有Runnable任務提交給它。 注意 - 在這種情況下,“大”是通過實驗和分析確定的:調整它直到您在速度和記憶方面的表現符合您的預期。
這是我所描述的一個例子:
// Using 10000 purely as a concrete example - you should define the correct number
public static final LARGE_NUMBER_OF_THREADS = 10000;
// Elsewhere in code, you defined a static thread pool
public static final ExecutorService EXECUTOR =
Executors.newFixedThreadPool(LARGE_NUMBER_OF_THREADS);
public void callBundles(final Map<String, Object> eventData) {
final Map<String, String> outputs =
(Map<String, String>)eventData.get(Constants.EVENT_HOLDER);
for (final BundleRegistration.BundlesHolderEntry entry : BundleRegistration.getInstance()) {
// "Three threads: one thread for the database writer,
// two threads for the plugin processors"
// so you'll need to repeat this future = E.submit() pattern two more times
Future<?> processFuture = EXECUTOR.submit(new Runnable() {
public void run() {
final Map<String, String> response =
entry.getPlugin().process(outputs);
//process the response and update database.
System.out.println(response);
}
}
// Note, I'm catching the exception out here instead of inside the task
// This also allows me to force order on the three component threads
try {
processFuture.get();
} catch (Exception e) {
System.err.println("Should really do something more useful");
e.printStackTrace();
}
// If you wanted to ensure that the three component tasks run in order,
// you could future = executor.submit(); future.get();
// for each one of them
}
為了完整起見,您還可以使用緩存的線程池來避免重復創建短期線程。 但是,如果您已經擔心內存消耗,那么固定池可能會更好。
當你進入Java 7時,你可能會發現Fork-Join比一系列Futures更好。 不過,最適合您的需求。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.