海量数据查询SOLR

Question

I have a provider that provide me the information using XML through SOLR Query. In order to extract the data I am using different url query with SOLR. The point is that I want to store more than 30.000.000 rows in our database.

In order to get all the data I am using this SOLR query

http://provider.com/query?q=colour%3A%20(red)&group.field=manufacturer&format=xml&group.limit=1 This query is providing me 1.311.707 results.

<products>
<grouped>
<matches>1311707</matches>
<groups>
<item>
<doclist>
<start>0</start>
<numFound>36242</numFound>
<docs>

If I want to extract all the data (matches) http://provider.com/query?q=colour%3A%20(red)&group.field=manufacturer&format=xml&group.limit=1311707

I receive this error:

<status_code>500</status_code>
<message/>
<error>Internal Server Error</error>

I am sure that is for the group.limit due to is too large.

How can I receive all the matches? How to query with SOLR large numbers of results?

Thank you very much

---

MIkpa

Using this request:

query?q=colour%3A%20(red)&fq=manufacturer:[%27%27%20TO%20*]&group=true&rows=1&start=1&group.field=manufacturer&group.offset=800&group.limit=8&format=xml

The value &start is increasing the groupValue, the &rows is how many groupValue is showed, &group.limit is the number of that appears in each query. If I try with &group.limit=160142 I receive Internal Server Error. So the point is, How can I have all the 160.142 ?

If I increase "group.offset=800" or 900 etc the value is always the same. How can i move the offset inside the ?

<products>
<grouped>
<matches>1311707</matches>
<groups>
<item>
<doclist>
<start>0</start>
<numFound>160142</numFound>
<docs>
<item>...</item>
<item>...</item>
<item>...</item>
<item>...</item>
<item>...</item>
<item>...</item>
<item>...</item>
<item>...</item>
</docs>
</doclist>
<groupValue>Fiat</groupValue>
</item>
</groups>

thank you

Answer 1

Try using the start and limit parameters to paginate result rows.

Try using group.offset and group.limit to paginate the result in the group.