FieldCache with frequently updating index

Question

Hi
I have lucene index that is frequently updating with new records, I have 5,000,000 records in my index and I\'m caching one of my numeric fields using FieldCache. but af

Answer 1

Here's one way I've solved this problem. You'll need to create a background thread to construct IndexSearcher instances, one at a time on some interval. Continue using your current IndexSearcher instance until a new one from the background thread is ready. Then swap out the new one to be your current one. Each instance acts as a snapshot of the index from the time that it was first opened. Note that the memory overhead for FieldCache doubles because you need two instances in memory at once. You can safely write to IndexWriter while this is happening.

If you need to you can take this a step further by making index changes immediately available for search, although it can get tricky. You'll need to associate a RAMDirectory with each snapshot instance above to keep the changes in memory. Then create a second IndexWriter that points to that RAMDirectory. For each index write you'll need to write to both IndexWriter instances. For searches you'll use a MultiSearcher across the RAMDirectory and your normal index on disk. The RAMDirectory can be thrown away once the IndexSearcher it was coupled with is no longer used. I'm glossing over some details here, however that's the general idea.

Hope this helps.