如何将一个just对象实例传递给java中的一个线程，这样即使实例值得到更新也不应该影响使用它的线程

Question

I am developing an application in which I continuously receive messages. I am storing these messages in an in-memory data structure (say List). I want to write those messages to a file but only after list size reaches some threshold value say 100 messages (perform batch processing on the messages). One way is I can simply check the list size after every message is received and call a function to write messages to a file if Threshold value is reached. But the problem with this approach is:

1. Calling function may need to wait indefinitely until all the messages are written to the file
2. The incoming messages may be lost in the process or might need to wait for getting stored in the List.

Other way could be to spawn a new thread, which will write messages to a file independently. But when I pass the list (containing messages) to the thread for performing write operation, it gets updated with new messages which are continuously coming in. As a result newly arrived messages also gets written to the file which is not expected.

This should not happen as I am intending the new messages to be written in the next batch.

Can someone suggest me a solution for this requirement, or any improvements in the above approach that can solve my issues.

Answer 1

I find a cleaner solution is to support auto-batching. ie where the size of the batch adjusts with the rate of incoming data.

To do this you can use a BlockingQueue

// unbound queue will not block the producer.
final BlockingQueue<T> queue = new LinkedBlockingQueue<T>();

// to add an element.
queue.add(element);

// to get a batch of data
List<T> list = new ArrayList<T>(maxElements);
while(writing) {
    T t = queue.take(); // wait for at least one element.
    list.add(t);
    queue.drainTo(list, maxElements-1);
    // process list, e.g. write to a file.
    list.clear();
}

The benefit of this approach is that if the producer is very slow, you won't get elements held unreasonably long, but as the rate increases the batch size naturally grows to whatever will keep up with the producer which means you don't have to decide what is the best batch size to use.

Answer 2

I would suggest the following approach:

1. Hold a reference to the message list in an AtomicReference .
2. When the list is full enough, replace it with a new, empty list;
3. pass the full list to a worker thread that will save the messages to a file.

If you write to the list from a single thread, it will be enough to use a plain reference instead of the AtomicReference .

Answer 3

It's important to understand that you never pass an object around in Java - only ever a reference (or a primitive value).

Options:

• Create a copy of the list, and pass a reference to that copy to your new thread
• Use a producer/consumer queue, so your "producing" thread only ever adds values to the queue, and your consumer thread only ever takes items from the queue to write them to disk. You'll want to think about how large you want the queue to potentially get before it stops accepting more entries, of course.

I'd recommend the latter approach, using the classes in the java.util.concurrent package to implement it; particularly BlockingQueue<E> implementations.

Answer 4

为什么主消息接收进程一旦将旧消息列表传递给文件写入线程就会创建新消息列表？

Answer 5

You can implement Custom BoundedQueue using Condition which accepts say 100 objects and then write at one go.

Now you can share this BoundedQueue class instance with different threads which will put objects in it and there will be thread which will call writeAll() method until you want to call it.

BoundedBuffer boundedBuffer  = new BoundedBuffer();
boundedBuffer.put("test"); .......

From writing thread do below

boundedBuffer.writeAll();

Below is the sample code

import java.util.concurrent.locks.Condition;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;

class BoundedBuffer {
final Lock lock = new ReentrantLock();
final Condition full = lock.newCondition();
final Condition empty = lock.newCondition();

final Object[] items = new Object[100];
int count;

public void put(Object x) throws InterruptedException {
    lock.lock();
    try {
        while (count == items.length) {
            empty.signal();
            full.await();
        }
        items[count] = x;
        ++count;
    } finally {
        lock.unlock();
    }
}

public void writeAll() throws InterruptedException {
    lock.lock();
    try {
        while (count < items.length)
            empty.await();
        // Write to file here After write finished signal full condition
        count = 0;
        full.signal();

    } finally {
        lock.unlock();
    }
}
}