How to configure Solr for improved indexing speed

Question

I have a client program which generates a 1-50 millions Solr documents and add them to Solr.
I\'m using ConcurrentUpdateSolrServer for pushing the documents from the client,

Answer 1

It looks like you are doing a bulk import of data into Solr, so you don't need to search any data right away.

First, you can increase the number of documents per request. Since your documents are small, I would even increase it to 100K docs per request or more and try.

Second, you want to reduce the number of times commits happen when you are bulk indexing. In your solrconfig.xml look for:

 
   15000
   false
 

You can disable autoCommit altogether and then call a commit after all your documents are posted. Otherwise you can tweak the numbers as follows:

The default maxTime is 15 secs so an auto commit happens every 15 secs if there are uncommitted docs, so you can set this to something large, say 3 hours (i.e. 3*60*60*1000). You can also add 50000000 which means an auto commit happens only after 50 million documents are added. After you post all your documents, call commit once manually or from SolrJ - it will take a while to commit, but this will be much faster overall.

Also after you are done with your bulk import, reduce maxTime and maxDocs, so that any incremental posts you will do to Solr will get committed much sooner. Or use commitWithin as mentioned in solrconfig.