POI single vs multithreaded performance

Question

I have a pretty simple use case - read an Excel spreadsheet with POI, populate some values, run cell updates and retrieve calculation outputs.

What puzzles me is performance. If I run single-threaded, Excel file takes a few seconds to get loaded/parsed, but for each subsequent request processing time decreases. If I run exactly the task via multiple threads and then join, the performance is actually 10 times slower on all threads. Where am I messing up?

Here is the test output:

Running Multi Threaded
Thread Thread-8 finished run 7092
Thread Thread-9 finished run 7092
Thread Thread-7 finished run 7092
Thread Thread-10 finished run 7092
Thread Thread-4 finished run 7092
Thread Thread-5 finished run 7092
Thread Thread-3 finished run 7107
Thread Thread-2 finished run 7107
Thread Thread-6 finished run 7107
Thread Thread-1 finished run 7108
Finished in 7113

Running Single Threaded
Thread Thread-11 finished run 591
Thread Thread-12 finished run 192
Thread Thread-13 finished run 173
Thread Thread-14 finished run 149
Thread Thread-15 finished run 126
Thread Thread-16 finished run 133
Thread Thread-17 finished run 159
Thread Thread-18 finished run 124
Thread Thread-19 finished run 131
Thread Thread-20 finished run 121
Finished in 1907
Process finished with exit code 0

Here is the test:

package com.test;

import org.apache.poi.xssf.usermodel.XSSFWorkbook;
import org.junit.jupiter.api.Test;

import java.io.BufferedInputStream;
import java.io.FileInputStream;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;

public class PerfTest {

    @Test
    public void runSingleThreaded() throws Exception {
        System.out.println("Running Single Threaded");

        long start = System.currentTimeMillis();
        for (int i = 0; i < 10; i++) {
            new PoiThread().run();
        }
        System.out.println("Finished in " + (System.currentTimeMillis() - start));
    }

    @Test
    public void runMultiThreaded() throws Throwable {
        System.out.println("Running Multi Threaded");

        List<Thread> threads = new ArrayList<>();
        for (int i = 0; i < 10; i++) {
            threads.add(new PoiThread());
        }

        long start = System.currentTimeMillis();
        threads.forEach(t -> t.start());
        threads.forEach(t -> {
            try {
                t.join();
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        });
        System.out.println("Finished in " + (System.currentTimeMillis() - start));
    }

    public static class PoiThread extends Thread {

        @Override
        public void run() {
            try {
                long runStart = System.currentTimeMillis();
                XSSFWorkbook workbook = new XSSFWorkbook(new BufferedInputStream(new FileInputStream("src/main/resources/customer.xlsx")));
                Helper helper = new Helper(workbook, "test", workbook.getCreationHelper().createFormulaEvaluator());
                String ref = helper.getFieldValueCellReference("Inputs", "customer");
                helper.evaluateAllCells();
                Map<String, Double> premiums = helper.getAllNumericFields("Premium Outputs");
                System.out.println(String.format("Thread %s finished run %s", this.getName(), (System.currentTimeMillis() - runStart)));
            } catch (Exception e) {
                e.printStackTrace();
            }
        }

    }

}

Answer 1

Huge thank you to all who've taken a look 🙏🙏🙏!

Looks like there are some caching/performance optimizations (JIT/C2?) happening at JVM/OS level that result in significant performance gains. Single threaded starts slow, and then picks up very quickly. Same for multi-threaded, each thread starts slow then picks up. If we start 10 threads - each starts with the worst performance, but then improves with subsequent runs.

So repeating each test 10 times (ie 100 files) will actually result in much better overall multi-threaded performance. It ends up being twice as fast as single threaded.

Answer 2

I don't believe your test case shows what you think it does.

Both of your methods, runSingleThreaded as well as runMultiThreaded , are running multi threaded. The only difference is that in your runMultiThreaded each thread uses Thread.join to wait for this thread to die. Furthermore your runMultiThreaded increases the memory overhead by storing all thread objects in a list in memory. So it is not really surprising that your runMultiThreaded performs worse.

Consider the following code:

import org.apache.poi.xssf.usermodel.XSSFWorkbook;

import java.io.BufferedInputStream;
import java.io.FileInputStream;

public class PerfTest {

    public void runSingleThreaded() throws Throwable {
        System.out.println("Running Single Threaded");

        long start = System.currentTimeMillis();
        for (int i = 0; i < 10; i++) {
            doTheWork("Work " + i);
        }
        
        System.out.println(String.format("runSingleThreaded finished in %s millis.", (System.currentTimeMillis() - start)));
    }

    public void runMultiThreaded() throws Throwable {
        System.out.println("Running Multi Threaded");

        long start = System.currentTimeMillis();
        Thread t = null;
        for (int i = 0; i < 10; i++) {
            t = new PoiThread();
            t.start();
        }
        
        // wait until last thread is finished
        if (t != null) t.join();
        
        System.out.println(String.format("runMultiThreaded finished in %s millis.", (System.currentTimeMillis() - start)));
    }
    
    public static void doTheWork(String workName) throws Throwable {
        long start = System.currentTimeMillis();
        BufferedInputStream in = new BufferedInputStream(new FileInputStream("./ExcelExampleIn.xlsx"));
        XSSFWorkbook workbook = new XSSFWorkbook(in);
        in.close();
        workbook.getCreationHelper().createFormulaEvaluator().evaluateAll();
        workbook.close();
        System.out.println(String.format("%s finished in %s millis.", workName, (System.currentTimeMillis() - start)));
    }

    public static class PoiThread extends Thread {
        @Override
        public void run() {
            try {
                doTheWork(this.getName());          
            } catch (Throwable e) {
                e.printStackTrace();
            }
        }
    }
    
    public static void main(String[] args) throws Throwable {
        //XSSFWorkbook workbook = new XSSFWorkbook(); workbook.close();
        
        PerfTest test = new PerfTest();
        test.runSingleThreaded();
        //test.runMultiThreaded();
    }
}

Here the runSingleThreaded really runs single threaded, means all in main thread. The runMultiThreaded starts threads and only waits for the last started thread to die.

If started using:

public static void main(String[] args) throws Throwable {
    //XSSFWorkbook workbook = new XSSFWorkbook(); workbook.close();
    
    PerfTest test = new PerfTest();
    test.runSingleThreaded();
    //test.runMultiThreaded();
}

It prints:

Running Single Threaded
Work 0 finished in 1850 millis.
Work 1 finished in 434 millis.
Work 2 finished in 396 millis.
Work 3 finished in 409 millis.
Work 4 finished in 351 millis.
Work 5 finished in 378 millis.
Work 6 finished in 368 millis.
Work 7 finished in 371 millis.
Work 8 finished in 318 millis.
Work 9 finished in 305 millis.
runSingleThreaded finished in 5199 millis.

Why the first work takes longer? This is because while first instantiating apache poi classes, as well as most other classes, super-ordinate tasks are running which need to run only once for each thread. For example to prepare the environment.

If started using:

public static void main(String[] args) throws Throwable {
    //XSSFWorkbook workbook = new XSSFWorkbook(); workbook.close();
    
    PerfTest test = new PerfTest();
    //test.runSingleThreaded();
    test.runMultiThreaded();
}

It prints:

Running Multi Threaded
Thread-2 finished in 3743 millis.
Thread-8 finished in 3747 millis.
Thread-5 finished in 3747 millis.
Thread-0 finished in 3747 millis.
Thread-9 finished in 3745 millis.
Thread-3 finished in 3745 millis.
Thread-7 finished in 3744 millis.
Thread-6 finished in 3744 millis.
Thread-1 finished in 3744 millis.
Thread-4 finished in 3743 millis.
runMultiThreaded finished in 3752 millis.

Here the super-ordinate tasks are running for each thread. But in whole it is faster than the single threaded processing.

Some of the super-ordinate tasks could be done before the main work starts. Using apache poi for example one could triggering the super-ordinate tasks, which will be in main thread only, by instantiating a main apache poi class.

If started using:

public static void main(String[] args) throws Throwable {
    XSSFWorkbook workbook = new XSSFWorkbook(); workbook.close();
    
    PerfTest test = new PerfTest();
    //test.runSingleThreaded();
    test.runMultiThreaded();
}

it prints:

Running Multi Threaded
Thread-0 finished in 2799 millis.
Thread-4 finished in 2805 millis.
Thread-2 finished in 2803 millis.
Thread-7 finished in 2800 millis.
Thread-5 finished in 2801 millis.
Thread-1 finished in 2800 millis.
Thread-8 finished in 2798 millis.
Thread-9 finished in 2797 millis.
Thread-6 finished in 2798 millis.
Thread-3 finished in 2800 millis.
runMultiThreaded finished in 2812 millis.

Here each single thread runs faster because the super-ordinate tasks, which are main thread only,, had run by instantiating the XSSFWorkbook class.