How to implement Lucene .Net search on Azure webrole

Question

I'm using AzureDirectory and Lucene .NET 2.9.4 but I have wo problems:

1. searcher doesn't seems to be so fast. I'm indexing with these settings: indexWriter.SetUseCompoundFile(false); indexWriter.SetMergeFactor(1000); index is around 3.5gb and it has 12.126.436 docs. To create the indexSearcher it takes around 5 min or more even if index is already on local disk. Is the index too big? I tried to perform a single term search using MultiFieldQueryParser on two fields. TermVector on fields is off
2. Everywhere is suggested to create only an instance of indexSearcher and share it between queries (in fact it is slow to be created) but I don't know how to share the Searcher singleton (it is the class that perform the search) between various web requests. If I create the singleton on the webrole class, then how can I use that instance to perform the search? At this moment every web requests recreates the singleton.

Thanks a lot

Answer 1

I have actually used that exact version of Lucene.NET with AzureDirectory and it doesn't work well. AzureDirectory in my opinion is not written for production scale.

If you look at the source code for AzureDirectory, it is using:

• older version of Lucene as a base (2.3x)
• exceptions are thrown everywhere (hard to debug/catch the right ones in production)
• it uses the old storage API (pre 1.8 version of the SDK)

I ended up creating my own dedicated Virtual Machine and using the .net 3.0.3 Lucene.Net library. Works like a champ in that environment, since I do not need to implement AzureDirectory.

You should have only ONE IndexWriter that is easy to implement with a storage queue. You can have multiple IndexReaders if you want to limit them write a IndexReader pool (like a SQL connection pool). I have multiple of those run fine with no exceptions flying around like they where with AzureDirectory.

My environment is a bit different lots of smaller indexes....not one massive one.

Answer 2

Maybe this is the AzureDirectory that people are talking about, maybe not - I tweaked this in order to get better performance. While I won't claim that it's production-grade and rock solid, it may help you over the AzureDirectory that you are currently using.

Hope it helps,