如何在Java + Akka上读取文件并发

Question

I have a big data csv file with key:value rows. How can i read it in parallel? I can not divide it to chunk because any row has another byte-size. What should i do in this case?

I can not find example on Java.

Answer 1

There is actually no reason to read the same file paralell, because that does NOT increase the speed of the file reading. If you want to read a file you have several options to do it:

1. You read the whole file to one byte[] at once, that's the fastest way of loading the file and after that you can split it to new lines and manage the data.

2. You read the lines from the file using a Scanner and the nextLine method. That's not really efficient, so I don't recommend this.

3. You read the file with some puffer byte array. That's a memory usage efficient solution, but the 1. option is still the best.

Also, because file loading is relatively slow (compared to data management in RAM) you should make a thread (yes, just only one, there is no need for more) which reads all the files to byte arrays, and maybe another thread which converts the byte[] to your loaded config, because that also can take lot of time if the file is big.