Solr索引方法和性能

Question

I am trying to understand how adding to a Solr index works and make sure that I am heading in a good direction.

The data set contains about 40000 NetCDF files, maybe 250KB each on average. For each file I need to index a subset of its metadata, and data.

{
'metadata' :
    {
    'file' : [id, date, ...],
    'identifiers' : [[a, b, c, ...]]
    },
'data' : 
    [[idx, time, lat, lon, a, b, c, ...]]
}

I have written a python script which calls a data subsetting web service using a few query strings, and generates a json object (with schema above) by filtering through all of the data. This is for a single file. Everything checks out here (although it could be faster).

My plan was to send this json object to Solr directly from the script, and this is where I have a few concerns:

-- I just created a ~160KB json file. I need to be memory concious, so I was wondering... do I have to keep this json object lying around in some file for solr to work? What happens if I generate a json file, index it, then delete the file?

-- Can I add the document from within the python script? I saw a few libraries that looked promising. I also recall from the documentation a way to send it to the solr url. If I must save the json file, can I make a system call to solr's post command, then delete the file after?

All I need the index to do is provide a url to the original NetCDF file, and the end user can use the index to gather the relevant info.

Does this sound reasonable? Any performance suggestions?

Answer 1

Irrespective of your indexing method, you are never required to keep the JSON files. You can certainly free up the space unless you want to re-index your data. Solr stores the all data internally as defined into schema. If you have fixed schema, then define into schema which all fields you want to store into solr and which all you want to just index. If you define all the fields as stored fields then you can always re-index even without source data. Here is sample field declaration. Please refer for more details. Link

<field name="firstname" type="string" indexed="true" stored="true" multivalued="false"/>