Too many open files Error on Lucene

Question

The project I\'m working on is indexing a certain number of data (with long texts) and comparing them with list of words per interval (about 15 to 30 minutes).

After som

Answer 1

You need to double check if ulimit value has actually been persisted and set to a proper value (whatever maximum is).

It is very likely that your app is not closing index readers/writers properly. I've seen many stories like this in the Lucene mailing list and it was almost always the user app which was to blame, not the Lucene itself.

Answer 2

I already used the ulimit but error still shows. Then I inspected the customized core adapters for lucene functions. Turns out there's too many IndexWriter.open directory that is LEFT OPEN.

Should note that after processing, will always call on closing the directory opened.

Answer 3

Use compound index to reduce file count. When this flag is set, lucene will write a segment as single .cfs file instead of multiple files. This will reduce the number of files significantly.

IndexWriter.setUseCompoundFile(true)