SOLR 4.1: Near Real time update/delete in SOLR doesn't work with searcher config

Question

I have my solr config work in near real time before. I recently added following config :

<listener event="firstSearcher" class="solr.QuerySenderListener">
  <arr name="queries">
    <lst> <str name="q">*:*</str>
          <str name="start">0</str>
          <str name="rows">10</str>
          <str name="facet">true</str>
          <str name="facet.field">ConversationId</str>              
          <str name="facet.mincount">1</str>
          <str name="facet.sort">count</str>              
          <str name="facet.method">fc</str>
    </lst>
    <lst> <str name="q">*:*</str>
          <str name="start">0</str>
          <str name="rows">10</str>
          <str name="facet">true</str>
          <str name="facet.field">UserName</str>              
          <str name="facet.method">fc</str>
    </lst>
  </arr>
</listener>
<listener event="newSearcher" class="solr.QuerySenderListener">
  <arr name="queries">
    <lst> <str name="q">*:*</str>
          <str name="start">0</str>
          <str name="rows">10</str>
          <str name="facet">true</str>
          <str name="facet.field">ConversationId</str>   
          <str name="facet.mincount">1</str>            
          <str name="facet.sort">count</str>              
          <str name="facet.method">fc</str>
    </lst>
    <lst> <str name="q">*:*</str>
          <str name="start">0</str>
          <str name="rows">10</str>
          <str name="facet">true</str>
          <str name="facet.field">UserName</str>              
          <str name="facet.method">fc</str>
    </lst>
  </arr>
</listener>

And it broke the NRT update. Basically, before the changes, I could add/delete a document and see the changes straight away but it stopped working after adding the above config and enable following filter cache:

<filterCache
  class="solr.LRUCache"
  size="1024"
  initialSize="512"
  autowarmCount="0"/>

Now when I add a document, I can see it but when I delete a document, seem like it doesn't want to disappear from the index. Apparently when I check the index in SOLR Admin, I see Last Modified : always > 1 minute. It was always 1 minute before.

So my question is: How do I maintain the Near Real Time search and still keep the searcher section above because I need it for fast facet search.

Here is my full config:

        <?xml version="1.0" encoding="UTF-8" ?>

    <config>

      <luceneMatchVersion>LUCENE_40</luceneMatchVersion>

       <dataDir>${data.dir}</dataDir>

      <lib dir="${lib.dir}" />

      <indexConfig>
        <writeLockTimeout>10000</writeLockTimeout>

        <ramBufferSizeMB>256</ramBufferSizeMB>

        <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
          <int name="maxMergeAtOnce">10</int>
          <int name="segmentsPerTier">10</int>
        </mergePolicy>

        <mergeFactor>10</mergeFactor>

        <lockType>native</lockType>

        <deletionPolicy class="solr.SolrDeletionPolicy">
          <str name="maxcommitstokeep">1</str>
          <str name="maxOptimizedCommitsToKeep">0</str>      
        </deletionPolicy>


         <infoStream file="${log.dir}">false</infoStream> 

      </indexConfig>


      <jmx />


      <directoryFactory name="DirectoryFactory" 
                        class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/> 

      <updateHandler class="solr.DirectUpdateHandler2">

        <updateLog>
        </updateLog>

        <autoCommit>
          <maxdocs>1000</maxdocs> <!-- maximum uncommited docs before autocommit triggered -->
          <maxTime>60000</maxTime> <!--    maximum time (in MS) after adding a doc before an autocommit is triggered. 60000 = 60sec -->
          <openSearcher>false</openSearcher>
        </autoCommit>

        <autoSoftCommit> 
           <maxTime>500</maxTime> 
        </autoSoftCommit>

      </updateHandler>

      <query>

        <filterCache
          class="solr.LRUCache"
          size="1024"
          initialSize="512"
          autowarmCount="0"/>


        <queryResultCache
          class="solr.LRUCache"
          size="10000"
          initialSize="512"
          autowarmCount="0"/>

        <documentCache
          class="solr.LRUCache"
          size="1024"
          initialSize="512"
          autowarmCount="0"/>

        <listener event="firstSearcher" class="solr.QuerySenderListener">
          <arr name="queries">
            <lst> <str name="q">*:*</str>
                  <str name="start">0</str>
                  <str name="rows">10</str>
                  <str name="facet">true</str>
                  <str name="facet.field">ConversationId</str>              
                  <str name="facet.mincount">1</str>
                  <str name="facet.sort">count</str>              
                  <str name="facet.method">fc</str>
            </lst>
            <lst> <str name="q">*:*</str>
                  <str name="start">0</str>
                  <str name="rows">10</str>
                  <str name="facet">true</str>
                  <str name="facet.field">UserName</str>              
                  <str name="facet.method">fc</str>
            </lst>
          </arr>
        </listener>
        <listener event="newSearcher" class="solr.QuerySenderListener">
          <arr name="queries">
            <lst> <str name="q">*:*</str>
                  <str name="start">0</str>
                  <str name="rows">10</str>
                  <str name="facet">true</str>
                  <str name="facet.field">ConversationId</str>   
                  <str name="facet.mincount">1</str>            
                  <str name="facet.sort">count</str>              
                  <str name="facet.method">fc</str>
            </lst>
            <lst> <str name="q">*:*</str>
                  <str name="start">0</str>
                  <str name="rows">10</str>
                  <str name="facet">true</str>
                  <str name="facet.field">UserName</str>              
                  <str name="facet.method">fc</str>
            </lst>
          </arr>
        </listener>

        <useFilterForSortedQuery>true</useFilterForSortedQuery>


        <queryResultWindowSize>100</queryResultWindowSize>
        <HashDocSet maxSize="3000" loadFactor="0.75"/>


        <boolTofilterOptimizer enabled="true" cacheSize="32" threshold=".05"/>

        <enableLazyFieldLoading>false</enableLazyFieldLoading>  

        <useColdSearcher>true</useColdSearcher>

      </query>

      <requestDispatcher handleSelect="true" >
        <requestParsers enableRemoteStreaming="true" multipartUploadLimitInKB="4096" formdataUploadLimitInKB="4096"/>
      </requestDispatcher>

      <httpCaching never304="true" />

      <requestHandler name="standard" class="solr.StandardRequestHandler" default="true" />

      <requestHandler name="/update" class="solr.UpdateRequestHandler">
          <lst name="defaults">
          <str name="update.chain">nondistributed</str>
        </lst>
      </requestHandler>


      <requestHandler name="/get" class="solr.RealTimeGetHandler">
         <lst name="defaults">
           <str name="omitHeader">false</str>
           <str name="wt">xml</str>
           <str name="indent">false</str>
         </lst>
      </requestHandler>               

      <requestHandler name="/admin/" class="solr.admin.AdminHandlers" />

      <requestHandler name="/admin/ping" class="solr.PingRequestHandler">
        <lst name="defaults">
          <str name="echoParams">all</str>
        </lst>

      </requestHandler> 

      <updateRequestProcessorChain name="nondistributed">
        <processor class="solr.LogUpdateProcessorFactory" />
        <processor class="solr.NoOpDistributingUpdateProcessorFactory"/>
        <processor class="solr.RunUpdateProcessorFactory" />
      </updateRequestProcessorChain>


      <admin>
        <defaultQuery>*:*</defaultQuery>
      </admin>

    </config>

Answer 1

1. Reduce the maxTime in autoCommit to more lower value (Suppose <maxTime>10000</maxTime> ). Also pass the commit statement with each update/delete.
2. Commit/Optimize your data daily (if possible) explicitly. Look into http://wiki.apache.org/solr/UpdateXmlMessages

Hope this might help.