Iterate over large collection in MongoDB via spring-data

Question

Friends!

I am using MongoDB in java project via spring-data. I use Repository interfaces to access data in collections. For some processing I need to iterate over all elements of collection. I can use fetchAll method of repository, but it always return ArrayList.

However, it is supposed that one of collections would be large - up to 1 million records several kilobytes each at least. I suppose I should not use fetchAll in such cases, but I could not find neither convenient methods returning some iterator (which may allow collection to be fetched partially), nor convenient methods with callbacks.

I've seen only support for retrieving such collections in pages. I wonder whether it is the only way for working with such collections?

Answer 1

Friends!

I am using MongoDB in java project via spring-data. I use Repository interfaces to access data in collections. For some processing I need to iterate over all elements of collection. I can use fetchAll method of repository, but it always return ArrayList.

However, it is supposed that one of collections would be large - up to 1 million records several kilobytes each at least. I suppose I should not use fetchAll in such cases, but I could not find neither convenient methods returning some iterator (which may allow collection to be fetched partially), nor convenient methods with callbacks.

I've seen only support for retrieving such collections in pages. I wonder whether it is the only way for working with such collections?

Answer 2

Friends!

I am using MongoDB in java project via spring-data. I use Repository interfaces to access data in collections. For some processing I need to iterate over all elements of collection. I can use fetchAll method of repository, but it always return ArrayList.

However, it is supposed that one of collections would be large - up to 1 million records several kilobytes each at least. I suppose I should not use fetchAll in such cases, but I could not find neither convenient methods returning some iterator (which may allow collection to be fetched partially), nor convenient methods with callbacks.

I've seen only support for retrieving such collections in pages. I wonder whether it is the only way for working with such collections?

Answer 3

Friends!

I am using MongoDB in java project via spring-data. I use Repository interfaces to access data in collections. For some processing I need to iterate over all elements of collection. I can use fetchAll method of repository, but it always return ArrayList.

However, it is supposed that one of collections would be large - up to 1 million records several kilobytes each at least. I suppose I should not use fetchAll in such cases, but I could not find neither convenient methods returning some iterator (which may allow collection to be fetched partially), nor convenient methods with callbacks.

I've seen only support for retrieving such collections in pages. I wonder whether it is the only way for working with such collections?

Answer 4

Friends!

I am using MongoDB in java project via spring-data. I use Repository interfaces to access data in collections. For some processing I need to iterate over all elements of collection. I can use fetchAll method of repository, but it always return ArrayList.

However, it is supposed that one of collections would be large - up to 1 million records several kilobytes each at least. I suppose I should not use fetchAll in such cases, but I could not find neither convenient methods returning some iterator (which may allow collection to be fetched partially), nor convenient methods with callbacks.

I've seen only support for retrieving such collections in pages. I wonder whether it is the only way for working with such collections?

Answer 5

Friends!

I am using MongoDB in java project via spring-data. I use Repository interfaces to access data in collections. For some processing I need to iterate over all elements of collection. I can use fetchAll method of repository, but it always return ArrayList.

However, it is supposed that one of collections would be large - up to 1 million records several kilobytes each at least. I suppose I should not use fetchAll in such cases, but I could not find neither convenient methods returning some iterator (which may allow collection to be fetched partially), nor convenient methods with callbacks.

I've seen only support for retrieving such collections in pages. I wonder whether it is the only way for working with such collections?

Answer 6

Friends!

I am using MongoDB in java project via spring-data. I use Repository interfaces to access data in collections. For some processing I need to iterate over all elements of collection. I can use fetchAll method of repository, but it always return ArrayList.

However, it is supposed that one of collections would be large - up to 1 million records several kilobytes each at least. I suppose I should not use fetchAll in such cases, but I could not find neither convenient methods returning some iterator (which may allow collection to be fetched partially), nor convenient methods with callbacks.

I've seen only support for retrieving such collections in pages. I wonder whether it is the only way for working with such collections?

Answer 7

Friends!

I am using MongoDB in java project via spring-data. I use Repository interfaces to access data in collections. For some processing I need to iterate over all elements of collection. I can use fetchAll method of repository, but it always return ArrayList.

However, it is supposed that one of collections would be large - up to 1 million records several kilobytes each at least. I suppose I should not use fetchAll in such cases, but I could not find neither convenient methods returning some iterator (which may allow collection to be fetched partially), nor convenient methods with callbacks.

I've seen only support for retrieving such collections in pages. I wonder whether it is the only way for working with such collections?

Answer 8

Friends!

I am using MongoDB in java project via spring-data. I use Repository interfaces to access data in collections. For some processing I need to iterate over all elements of collection. I can use fetchAll method of repository, but it always return ArrayList.

However, it is supposed that one of collections would be large - up to 1 million records several kilobytes each at least. I suppose I should not use fetchAll in such cases, but I could not find neither convenient methods returning some iterator (which may allow collection to be fetched partially), nor convenient methods with callbacks.

I've seen only support for retrieving such collections in pages. I wonder whether it is the only way for working with such collections?

Answer 9

This answer is based on: https://stackoverflow.com/a/22711715/5622596

That answer needs a bit of an update as PageRequest has changed how it is being constructed.

With that said here my modified response:

int pageNumber = 1;

//Change value to whatever size you want the page to have
int pageLimit = 100;

Page<SomeClass> page;
List<SomeClass> compondList= new LinkedList<>();

do{
    PageRequest pageRequest = PageRequest.of(pageNumber, pageLimit);
    
    page = repository.findAll(pageRequest);
    
    List<SomeClass> listFromPage = page.getContent();

    //Do something with this list example below
    compondList.addAll(listFromPage);

    pageNumber++;

  }while (!page.isLast());

//Do something with the compondList: example below
return compondList;