Creating a concurrent collection which does not load data in memory from Stream : Java

Question

I have a requirement to send each line from a Huge file to a Java Collection,Iterator,Iterable. From this collection,iterator etc an existing framework takes the data and loads into a processing system. However the requirement is to send a Concurrent Collection,iterator etc to this framework, this is the part i am facing difficulty.

I have a java.util.stream which i got from a huge file. There are numerous lines in the file, i want to pass all the lines into a java Concurrent java.lang.Iterable or a Concurrent java.util.Iterator or a Concurrent java.util.Collection which does not store all the records in the memory so that i do not get Exceptions like OutOfMemory etc.,

• I was able to get a Iterator implementation from the java Stream but it is not Concurrent

. Any idea on how to make a Concurrent implementation with or without using the Stream API will will be much appreciated.

Answer 1

There is no such thing as a “Concurrent Iterator”.

While there are Concurrent Collections , the Iterator s created for them are weakly consistent (which describes their behavior regarding updates of the underlying Collection ) and a single Iterator instance can only be used by a one thread at a time.

Since the Iterator from a Stream created via Files.lines has no underlying Collection that could get updated and doesn't support the remove() operation, it can't exhibit any behavior that would contradict the behavior of an Iterator created for a Concurrent Collection, apparently expected by the framework¹. Well, except that it doesn't support the remove operation, but I have the feeling that this is not an issue for your desired operation.

So if the framework offers you the option to pass an Iterator to it, you might use it to pass the Iterator created via Stream.iterator() . As long as the Stream doesn't have either of the stateful intermediate operations sort or distinct , it will exhibit the desired lazy behavior.

---

¹ Most notably, it will never throw a ConcurrentModificationException .