来自Iterator的Java ExecutorService读取任务

Question

All,

I'm using a Java ExecutorService to perform tasks in parallel. Unfortunately, the list of tasks is now reaching the tens of millions. This means that submitting the tasks to the executor service ahead of time is infeasible due to memory constraints.

I am able to generate an iterator which dynamically creates the tasks as they are needed, but I'm not sure how best to apply this to the ExecutorService.

Should I create a task which pulls the next task from the iterator or is there some better way to do this?

Answer 1

A quick experiment produced this that kind of works. It certainly should demonstrate one way of doing it.

I create and run a ServiceFeeder which delivers Runnables to the service via the execute method.

ExecutorService service = Executors.newFixedThreadPool(10);

class ServiceFeeder implements Runnable {
    final Iterator<Runnable> i;

    public ServiceFeeder(Iterator<Runnable> i) {
        this.i = i;
    }

    @Override
    public void run() {
        while (i.hasNext()) {
            service.execute(i.next());
        }
    }
}

public void test() throws Exception {
    System.out.println("Hello world!");
    // Demo Iterator<Runnable> - use yours.
    Iterator<Runnable> i = new Iterator<Runnable>() {
        volatile int n = 0;

        @Override
        public boolean hasNext() {
            return n < 100;
        }

        @Override
        public Runnable next() {
            return () -> System.out.println(n++);
        }
    };

    ServiceFeeder feeder = new ServiceFeeder(i);
    Thread feederThread = new Thread(feeder);
    feederThread.start();
    // Wait for the feeder to stop.
    feederThread.join();
    // Wait for the service to stop.
    service.shutdown();
}

This kind of works because it prints far more than I expected but that's not a problem as a demo IMHO.