solr delta-import “fetches” all data before “processing”?

Question

I use solr's DIH to delta import data from mysql to solr, but i found when the table has a large number of records (100000000), the delta-import need to fetch all the data before processing . So it always cause a large memory consumption resulting in Running Out of Memory . While when i do a full import operation, it may fetch a part of the data.

Is there any setting on the solr delta import to fix this problem?

Answer 1

One solution might be to set the batchSize in your data-config.xml . Here's an example on how to do it:

<dataSource type="JdbcDataSource" name="ds-2" driver="com.mysql.jdbc.Driver" url="jdbc:mysql://localhost:8889/mysqldatabase" batchSize="-1" user="root" password="root"/>

This example is taken from the following link: DataImportHandler - Solr Wiki

Another thing you might try is using the full-import command for your delta, but setting the batchSize has worked for me in the past. If you'd like to try using the full-import for your delta, visit the following link: Full Import Delta - Solr Wiki