speed of importing data from another solr is very slow

Question

I was using SolrEntityProcessor to import data from another solr. It was working but the speed is only 30 documents/s which is weigh below I was importing from SQL Sever (7000 documents/s).

I am wondering if I was missing something and if there is any way to improve the speed.

A litte information about my solr cloud. I have 3 shards on three different servers and all of them are within same domain.

Thanks in advance.

Answer 1

Quoting from SolrEntityProcessor gets slower and slower

SolrEntityProcessor is fine for small amounts of data but not useful for such a large index. The problem is that deep paging in search results is expensive. As the "start" value for a query increases so does the cost of the query. You are much better off just re-indexing the data.

To resolve this:

The common way to deal with this issue is by filtering on a value that would return fewer results every query, as a creation_date field, and every query change this field range. For your data import use-case you might want to generate your data-import.xml with different entities, each one for another creation_date range. Thus no need for deep paging.