如何减少Solr中多值字段的长度

Question

We have a multivalued field in Solr that we want to reduce its length. A sample result response is as follows:

 response": {
     "numFound": 1,
     "start": 0,
     "docs": [
       {
         "created_date": "2016-11-23T13:47:46.55Z",
         "solr_index_date": "2016-12-01T08:21:59.78Z",
         "modified_date": "2016-12-13T08:45:44.507Z",        
         "id": "FEAE38C2-ABFF-4F0C-8AFD-9B8F51036D8A",        
         "Field1": [
           "false",
           "true",
           "true",
           .....   <= 1200 items
         ]
       }
         ]   
     }

We have big data, a couple of TB and we are looking for an optimized way to alter all documents within Solr and to modify Field1 to contain only the first 100 items.

Can something like this be done without the need to write a script to manually fetch the document, make adjustments and push it back to solr? Has anyone had a similar experience? Thanks

Answer 1

We have faced this problem. But we use Two collections to solve this problem. Use SoleEntityProcessor to move the document from one collection to another.

[SolrEntityProcessor]

<dataConfig>
  <document>
    <entity name="sep" processor="SolrEntityProcessor" url="http://localhost:8983/solr/db" query="*:*"/>
  </document>
</dataConfig>

While moving pass that document through updateRequestProcessorChain where we can write StatelessScriptUpdateProcessorFactory to edit our documents or to truncate the multivalued field.
In StatelessScriptUpdateProcessorFactory you can get the field and apply your operations and then reset that field.

[StatelessScriptUpdateProcessorFactory]

function processAdd(cmd) {
    doc = cmd.solrDoc;
    multiDate = doc.getFieldValue("multiValueField");
    //Apply your operation to above field
    //doc.setField("multiValueField",value);

}
function processDelete(cmd) {
  // no-op
}

function processMergeIndexes(cmd) {
  // no-op
}

function processCommit(cmd) {
  // no-op
}

function processRollback(cmd) {
  // no-op
}

function finish() {
  // no-op
}

For More information on StatelessScriptUpdateProcessorFactory, you can refer to this question On solr how can i copy selected values only from multi valued field to another multi valued field? in which they edit the multivalued field using the script.