How to maintain Sitecore Lucene indexes in huge content delivery webfarm?

Question

I know that Lucene indexes cannot be shared:

Sitecore keeps a local copy of the Lucene index in the file system of each instance and does not suppor

Answer 1

You can also consider using the open source Sitecore Lucene Refresher that will run a index crawl operation in-memory and will commit the index back to the file system so you don't lose any index content during the rebuild process. This can at least help. Then maybe, set up some sort of agent to run this crawl/rebuild operation at a specific time of day to keep all CD servers doing this at the same time in-sync.

Answer 2

It should probably be pointed out that sitecore now recommends that you use Solr in this scenario and not to try and synchonise multiple Lucene indexes:

The general reasons for using Solr instead of Lucene are...

If you use multiple content delivery servers (or plan to do so later), use Solr. Solr works automatically in such an environment. You could use Lucene, but you have to make sure that indexes are synchronized across servers yourself.

You should therefore use Solr if you plan to scale your site (have a distributed setup with multiple servers).

From Using Solr or Lucene

Answer 3

Wesley Lomax answer is correct. However, I want to point out that I was also involved in the same situation where I have items in my Data Folder in 1000's. I updated my web.config setting like this:

 <!--  INDEX FOLDER
        The path to the folder where the Lucene.Net search indexes are stored.
        Default value: $(dataFolder)/indexes
  -->
  <setting name="IndexFolder" value="$(dataFolder)/indexes" />
  <!--  INDEX UPDATE INTERVAL
        Gets the interval between the IndexingManager checking its queue for pending actions.
        Default value: "00:01:00" (1 minute)
  -->
  <setting name="Indexing.UpdateInterval" value="00:00:30" />
  <!--  INDEX UPDATE JOB THROTTLE
        Gets the minimum time to wait between individual index update jobs.
        Default value: "00:00:01" (1 second)
  -->
  <setting name="Indexing.UpdateJobThrottle" value="00:00:01" />

Answer 4

You need to enable History Engine for the CM and CD servers web database.

See this extract from the Sitecore Scaling Guide.

To enable History Engine for a Sitecore database: In the web.config file, add the following section to the /configuration/sitecore/databases/database element, where id equals the name of the database:

<Engines.HistoryEngine.Storage>
  <obj type="Sitecore.Data.$(database).$(database)HistoryStorage, Sitecore.Kernel">
   <param connectionStringName="$(id)" />
   <EntryLifeTime>30.00:00:00</EntryLifeTime>
  </obj>
</Engines.HistoryEngine.Storage>
<Engines.HistoryEngine.SaveDotNetCallStack>false</Engines.HistoryEngine.SaveDotNetCallStack>

When a Sitecore item is changed, the Lucene indexes are updated immediately on the Sitecore instance where the change was made. On remote servers in a multi-server environment, the Lucene indexes are not updated immediately after an item is changed. The Lucene indexes are automatically updated after the interval that is defined in the web.config file, in the Indexing.UpdateInterval setting and with the minimum wait time between the two consequent updates defined in the Indexing.UpdateJobThrottle setting.

See here