solr delta import “fetches” but does not “process”

Question

when I run /solr/dataimport?command=full-import it processes all the documents. However, when I run a delta import (/solr/dataimport?command=delta-import) it correctly identifies the updated data (returns " <str name="Total Rows Fetched">1</str> ") but does not process any of it (returns " <str name="Total Changed Documents">0</str> ")

my data-config.xml looks something like this:

<entity name="category" pk="catID" 
query="SELECT CONCAT('c_',catID) ID, catID, catName FROM category" 
deltaImportQuery="SELECT CONCAT('c_',catID) ID, catID, catName FROM category WHERE catID = '{$dataimporter.delta.catID}'" 
deltaQuery="SELECT catID FROM category WHERE catDate &gt; '${dataimporter.last_index_time}'"/>

(note - there is a seperate reason for my concat)

why does the full import process while the delta import fetches but does not process?

Answer 1

'{$dataimporter.delta.catID}' , here the catID should be the value of the name attribute in the field definitions. I had same problem, then realized that the referenced variable is the field name... I hope it works.

Answer 2

I had same issue and figured out that deltaImportQuery is case sensitive

Made my id Column as "ID"

deltaImportQuery="select id,state,name,place,city from temp where ID='${dih.delta.ID}

Answer 3

This is because a full-import will process every document in your database.

A delta-import only processes documents that have been modified since your last import of any kind (in your case, dictated by the catDate field).

For example, suppose you do a full-import at 2012-09-09 12:15:38.

Next, you do a delta-import at 2012-09-10 12:15:38. Only documents that have a catDate > '2012-09-09 12:15:38' will be processed.

The idea is that delta-import is a way to maintain an ever changing index without having to do a full-import each time.

Answer 4

In your case, the delta query works fine but your delta-import query fails.

This is because you have given,

deltaImportQuery="SELECT CONCAT('c_',catID) ID, catID, catName FROM category WHERE catID = '{$dataimporter.delta.catID}' "

But it should be given as,

deltaImportQuery="SELECT CONCAT('c_',catID) ID, catID, catName FROM category WHERE catID = '${dataimporter.delta.catID}' "