如何在Java中使用多个线程迭代一个Collection，其中没有两个线程迭代在Collection的同一部分？

Question

I need to iterate over a large ArrayList (~50,000 entries) and I need to use multiple threads to do this fairly quickly.

But I need each thread to start at a unique index so that no two threads ever iterate over the same part of the list. There will be a batchSize of 100 so each thread will loop from its startIndex to startIndex + 100 .

Is there any way to achieve this? Note that I am only performing read operations here, no writes. Each entry in the list is just a String which is actually an SQL query which I am then executing against a DB via JDBC.

Answer 1

If you only intend to read the List , not mutate it, you can simply define your Runnable to take the List and a startIndex as constructor arguments. There's no danger to concurrently reading an ArrayList (even the same indices) as long as no threads modify it at the same time.

To be safe, be sure to wrap your ArrayList in a call to Collections.unmodifiableList() and pass that List to your Runnable s. That way you can be confident the threads will not modify the backing ArrayList .

Alternatively, you can construct sublists in your main thread (with List.subList() ) so that you don't need to pass the startIndex to each thread. However you still want to make the sublists unmodifiable before you do so. Six of one, half a dozen of the other.

Even better would be to use Guava 's ImmutableList ; it's naturally thread-safe.

There's also parallel streams in Java 8, but take care with this solution; they're powerful, but easy to get wrong.

Answer 2

If you use Java 8, look at list.stream().parallel()

For Java 7, use subList() outside of the threads to split the work into pieces. The threads should then just operate on such a sub-list. For most lists, subList() is a very efficient operation which doesn't copy the data. If the backing list is modified, then you get a ConcurrentModificationException

As the pumping the data to the threads, I suggest to look at the Executor API and Queue s. Just put all the work pieces in the queue and let the executor figure everything out.

Answer 3

Have an atomic variable:

int nextBatch = 0;

Increment it every time a thread concumes a new batch:

public synchronized int getNextBatch() {
    nextBatch += batchSize;
    if(nextBatch >= arraylist.size()) {
        // The end was reached
        return -1;
    }
    return nextBatch;
}

A thread will call this method and get the the range we will need to work on:

int start = getNextBatch();
if(start == -1) {
    // The end was reached
}
int end = Math.min(start + batchSize, arraylist.size);

// Iterate over its own range
for(int i = start; i < end; i++) {
    Object obj = arraylist.get(i);
    // Do something with obj
}