Apache Solr: bitwise operations to filter search results

Question

I need to filter solr search results corresponding the access rights in our cms (drupal 7 + custom access control mechanism based on bitmasks).

There exists a Solr QParserPlugin plugin for search results filtering based on bitwise operations on integer fields: https://issues.apache.org/jira/browse/SOLR-1913 .

I am using Solr 3.6.1 (+ the plugin in /var/lib/tomcat6/solr/lib/bitwise_filter_plugin.jar) on tomcat6 (on debian system) with the schema.xml provided by the drupal module search_api_solr and the solrconfig.xml found in the modules issue queue (extended as indicated in the SOLR-1913 issue).

The Solr query ...

http://solr:8080/solr/select?qf=t_title&fl=*,score&fq={!bitwise field=is_bitmask op=AND source=1234}*

... fails with the following message in the error log:

Sep 27, 2012 8:57:41 AM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/select params={qf=t_title&fl=*,score&fq={!bitwise+field%3Dis_bitmask+op%3DAND+source%3D1234}} status=500 QTime=15 
Sep 27, 2012 8:57:41 AM org.apache.solr.common.SolrException log
SEVERE: java.lang.NullPointerException
  at org.apache.lucene.search.FilteredQuery.hashCode(FilteredQuery.java:268)
  at java.util.AbstractList.hashCode(AbstractList.java:542)
  at org.apache.solr.search.QueryResultKey.<init>(QueryResultKey.java:49)
  at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1084)
  at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:375)
  at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:394)
  at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:186)
  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
  at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376)
  at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365)
  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
  at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
  at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
  at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
  at java.lang.Thread.run(Thread.java:636)

The only thing I can see in the error message is that the plugin is invoked by solr.

Any help would be appreciated, Thanks.

Answer 1

Instead of making Solr do it your way, you might want to try it Solr's way. Solr already implements boolean logic.

Split your bitfield into a set of named boolean fields. They can even be bit_0, bit_1, etc. if you are short on inspiration. You can use a dynamic field to save typing and allow later expansion.

Index them as bit_0:true and so on for each doc.

Use those fields in the filter query to do selection: bit_0:true AND bit_24:true.

This is likely to run much, much faster than a bitwise comparison on each field. The bitwise comparise probably needs a full table scan on each query. Well, full field value scan, since Solr does not have tables.