How to make sure Solr/Lucene won't die with java.lang.OutOfMemoryError?

Question

I\'m really puzzled why it keeps dying with java.lang.OutOfMemoryError during indexing even though it has a few GBs of memory.

Is there a fundamental reason why it needs

Answer 1

An old question but since I stumbled upon it:

1. The String Field Cache is lot more compact from Lucene 4.0. So lot can fit in.
2. Field Cache is an in-memory structure. So can't prevent OOME.
3. For fields which need sorting or faceting - one should try DocValues to overcome this problem. DocValues do work with numeric and non-analyzed string values. And I presume many use cases of sorting/faceting will have one of these value types.

Answer 2

I'm not certain there is a steadfast way to ensure you won't run into OutOfMemoryExceptions with Lucene. The problem you are facing is problem related to the use of FieldCache. From the Lucene API "Maintains caches of term values.". If your terms exceed the amount of memory allocated to the JVM you'll get the exception.

The documents are being sorted "at org.apache.lucene.search.FieldComparator$StringOrdValComparator.setNextReader(FieldComparator.java:667)", which will take up as much memory as is needed to store the terms being sorted for the index.

You'll need to review projected size of the fields that are sortable and adjust the JVM settings accordingly.