多线程-内存不足

Question

I am using an ThreadPoolExecutor with 5 active threads, number of tasks is huge 20,000.
The queue is filled up ( pool.execute(new WorkingThreadTask()) ) with instances of a Runnable tasks almost immediately.

Each WorkingThreadTask has a HashMap :

Map<Integer, HashMap<Integer, String>> themap ;

each map can have up to 2000 items, and each sub-map has 5 items. There is also a shared BlockingQueue .

When process is running I am getting out of memory. I'm running with: (32bit -Xms1024m -Xmx1024m)

How can I handle this problem? I don't think I have leaks in hashmap... When the thread is finished hashmap is cleaned right?

Update:

After running a profiler and checking the memory, the biggest hit is:

byte[] 2,516,024 hits, 918 MB  

I don't know from where it's called or used.

Name    Instance count  Size (bytes)
byte[ ] 2519560 918117496
oracle.jdbc.ttc7.TTCItem    2515402 120739296
char[ ] 357882  15549280
java.lang.String    9677    232248
int[ ]  2128    110976
short[ ]    2097    150024
java.lang.Class 1537    635704
java.util.concurrent.locks.ReentrantLock$NonfairSync    1489    35736
java.util.Hashtable$Entry   1417    34008
java.util.concurrent.ConcurrentHashMap$HashEntry[ ] 1376    22312
java.util.concurrent.ConcurrentHashMap$Segment  1376    44032
java.lang.Object[ ] 1279    60216
java.util.TreeMap$Entry 828 26496
oracle.jdbc.dbaccess.DBItem[ ]  802 10419712
oracle.jdbc.ttc7.v8TTIoac   732 52704

Answer 1

I'm not sure about the inner map but I suspect the problem is that you are creating a large number of tasks that is filling memory. You should be using a bounded task queue and limit the job producer.

Take a look at my answer here: Process Large File for HTTP Calls in Java

To summarize it, you should create your own bounded queue and then use a RejectedExecutionHandler to block the producer until there is space in the queue. Something like:

final BlockingQueue<WorkingThreadTask> queue =
    new ArrayBlockingQueue<WorkingThreadTask>(100);
ThreadPoolExecutor threadPool =
    new ThreadPoolExecutor(nThreads, nThreads, 0L, TimeUnit.MILLISECONDS, queue);
// we need our RejectedExecutionHandler to block if the queue is full
threadPool.setRejectedExecutionHandler(new RejectedExecutionHandler() {
       @Override
       public void rejectedExecution(WorkingThreadTask task,
             ThreadPoolExecutor executor) {
           try {
                // this will block the producer until there's room in the queue
                executor.getQueue().put(task);
           } catch (InterruptedException e) {
                throw new RejectedExecutionException(
                   "Unexpected InterruptedException", e);
           }
    }
});

Edit:

I don't think I have leeks in hashmap... when thread is finished hashmap is cleaned right?

You might consider aggressively calling clear() on the work HashMap and other collections when the task completes. Although they should be reaped by the GC eventually, giving the GC some help may solve your problem if you have limited memory.

If this doesn't work, a profiler is the way to go to help you identify where the memory is being held.

Edit:

After looking at the profiler output, the byte[] is interesting. Typically this indicates some sort of serialization or other IO. You may also be storing blobs in a database. The oracle.jdbc.ttc7.TTCItem is very interesting however. That indicates to me that you are not closing a database connection somewhere. Make sure to use proper try/finally blocks to close your connections.

Answer 2

HashMap carries quite a lot of overhead in terms of memory usage..... it carries about 36 bytes minimum per entry, plus the size of the key/value itself - each will be at least 32 bytes (I think that's about the typical value for 32-bit sun JVM).... doing some quick math:

20,000 tasks, each with map with 2000 entry hashmap. The value in the map is another map with 5 entries.
->  5-entry map is 1* Map + 5* Map.Object entries + 5*keys + 5*values = 16 objects at 32 bytes => 512 bytes per sub-map.
->  2000 entry map is 1* Map, 2000*Map.Object + 2000 keys + 2000 submaps (each is 512 bytes) => 2000*(512+32+32) + 32 => 1.1MB
->  20,000 tasks, each of 1.1MB -> 23GB

So, your overall footprint is 23GB.

The logical solution is to restrict the depth of your blocking queue feeding the ExecutorService, and only create enough child tasks to keep it busy..... set a limit of about 64 entries in the queue, and then you will never have more than 64 + 5 tasks instantiated at one time. When wpace comes available in the executor's queue, you can create and add another task.

Answer 3

You can improve the efficiency by not adding so many tasks ahead of what is being processed. Try checking the queue and only adding to it if there is less than 1000 entries.

You can also make the data structures more efficient. A Map with an Integer key can often be reduced to an array of some kind.

Lastly, 1 GB isn't that much these days. My mobile phone has 2 GB. If you are going to process large amount of data, I suggest getting a machine with 32-64 GB of memory and a 64-bit JVM.

Answer 4

From the large byte[] s, I'd suspect IO related issues (unless you are handling video/audio or something).

Things to look at:

• DB: Are you trying to read large amount of stuff at once? You can eg use a cursor to not do that
• File/Network: Are you trying to read large amounts of stuff from file/network at once? You should "propagate the load" to whatever is reading and regulate the rate of read.

UPDATE : OK, so you are using a cursor to read from DB. Now you need to make sure that the reading from the cursor only progresses as you finish stuff (aka "propagate the load"). To do this, use a thread pool like this:

 BlockingQueue<Runnable> queue = new LinkedBlockingQueue<Runnable>(queueSize);
 ThreadPoolExecutor tpe = new ThreadPoolExecutor(
                    threadNum,
                    threadNum,
                    1000,
                    TimeUnit.HOURS,
                    queue,
                    new ThreadPoolExecutor.CallerRunsPolicy());

Now when you post to this service from your code which reads from the DB, it will block when the queue is full (the calling thread is used to run tasks and hence blocks).