elasticsearch JDBC -RIVER java.lang.OutOfMemoryError: unable to create new native thread

问题

I am using elasticsearch "1.4.2" with river plugin on an aws instance with 8GB ram.Everything was working fine for a week but after a week the river plugin[plugin=org.xbib.elasticsearch.plugin.jdbc.river.JDBCRiverPlugin version=1.4.0.4] stopped working also I was not able to do a ssh login to the server.After server restart ssh login worked fine ,when I checked the logs of elastic search I could find this error.

[2015-01-29 09:00:59,001][WARN ][river.jdbc.SimpleRiverFlow] no river mouth
[2015-01-29 09:00:59,001][ERROR][river.jdbc.RiverThread   ] java.lang.OutOfMemoryError: unable to create new native thread
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: unable to create new native thread

After restarting the service everything works normal .But after certain interval the same thing happen.Can anyone tell what could be the reason and solution .If any other details are required please let me know.

When I checked the number of file descriptor using

sudo ls /proc/1503/fd/ | wc -l

I could see it is increasing after every time . It was 320 and it now reached 360 (keeps increasing) . and

sudo grep -E "^Max open files" /proc/1503/limits

this shows 65535

processor info

vendor_id   : GenuineIntel
cpu family  : 6
model       : 62
model name  : Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
stepping    : 4
microcode   : 0x415
cpu MHz     : 2500.096
cache size  : 25600 KB
siblings    : 8
cpu cores   : 4

memory

MemTotal:       62916320 kB
MemFree:        57404812 kB
Buffers:          102952 kB
Cached:          3067564 kB
SwapCached:            0 kB
Active:          2472032 kB
Inactive:        2479576 kB
Active(anon):    1781216 kB
Inactive(anon):      528 kB
Active(file):     690816 kB
Inactive(file):  2479048 kB

回答1:

Do the following

Run the following two commands as root:

ulimit -l unlimited
ulimit -n 64000
In /etc/elasticsearch/elasticsearch.yml make sure you uncomment or add a line that says:

bootstrap.mlockall: true
In /etc/default/elasticsearch uncomment the line (or add a line) that says MAX_LOCKED_MEMORY=unlimited and also set the ES_HEAP_SIZE line to a reasonable number. Make sure it's a high enough amount of memory that you don't starve elasticsearch, but it should not be higher than half the memory on your system generally and definitely not higher than ~30GB. I have it set to 8g on my data nodes.

In one way or another the process is obviously being starved of resources. Give your system plenty of memory and give elasticsearch a good part of that.

回答2:

I think you need to analysis your server log. Maybe In: /var/log/message

来源：https://stackoverflow.com/questions/28556125/elasticsearch-jdbc-river-java-lang-outofmemoryerror-unable-to-create-new-nativ

标签

ElasticSearch

elasticsearch-jdbc-river