Java-使用大型Excel进行读取，处理和写入

Question

I have a large spread sheet. It has 10 sheets, each with 1m rows. With Java, I need to run an algorithm for each row, return a value for each row and insert back into the excel file.

My idea was to load the file into ram, do calculations for each row, store the result in a list, and insert back to excel in order, but I didn't anticipate the issues dealing with the data size.

I tried XSSF, and it wasn't able to load such a large file. After waiting for a few hours it gave me the OOM error.

I tried increasing heap in run->run configurations->arguments, and in control panel->java. It didn't work.

I tried using the following StreamingReader and it didn't work.

FileInputStream in = new FileInputStream("D:\\work\\calculatepi\\sampleresult.xlsx");
Workbook workbook = StreamingReader.builder()
    .rowCacheSize(100)  
    .bufferSize(4096)  
    .open(in);  

I'm really out of clue and not sure what to do. Is there no easy way to do this?

Answer 1

It is not only about the configuration of that library. It is also about the memory that you give to you JVM! Try increasing the heap space of the JVM, see here for example.

Beyond that: I think you should do two things:

• make experiments with smaller sheets. Create one that only has 100 rows, then maybe 10K, 100K. Measure the memory consumption. And from there
• see if there are other APIs/libraries that allow you to read/write individual rows without pulling the whole file into memory
• and if none of that works, maybe you have to use a completely different design: such as just having some sort of "service". And now, you write some VB script code that you run inside excel, that simply for each row calls that service to fetch the results. Or, ideally: do not misuse Excel as database. This is similar to using a sports car to transport a huge number of goods, just because you already have that sports car. But it would still be more appropriate to get yourself a truck instead. In other words: consider moving your data into a real database. In the long run, everything you do will be "easier" then!