elasticsearch 6.8.10 集群报错处理

妖精的绣舞 提交于 2020-08-11 22:56:56

报错如下

[2020-06-24T16:19:54,728][WARN ][o.e.b.JNANatives         ] [8.1.10.146] Unable to lock JVM Memory: error=12, reason=无法分配内存
[2020-06-24T16:19:54,731][WARN ][o.e.b.JNANatives         ] [8.1.10.146] This can result in part of the JVM being swapped out.
[2020-06-24T16:19:54,731][WARN ][o.e.b.JNANatives         ] [8.1.10.146] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2020-06-24T16:19:54,731][WARN ][o.e.b.JNANatives         ] [8.1.10.146] These can be adjusted by modifying /etc/security/limits.conf, for example: 
	# allow user 'elasticsearch' mlockall
	elasticsearch soft memlock unlimited
	elasticsearch hard memlock unlimited
[2020-06-24T16:19:54,732][WARN ][o.e.b.JNANatives         ] [8.1.10.146] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2020-06-24T16:19:54,879][INFO ][o.e.e.NodeEnvironment    ] [8.1.10.146] using [1] data paths, mounts [[/data (/dev/mapper/datavg-datalv)]], net usable_space [294.8gb], net total_space [294.8gb], types [xfs]
[2020-06-24T16:19:54,879][INFO ][o.e.e.NodeEnvironment    ] [8.1.10.146] heap size [29.4gb], compressed ordinary object pointers [true]
[2020-06-24T16:19:54,880][INFO ][o.e.n.Node               ] [8.1.10.146] node name [8.1.10.146], node ID [LQzip03MR9a_gCEp5xLdhw]
[2020-06-24T16:19:54,881][INFO ][o.e.n.Node               ] [8.1.10.146] version[6.8.10], pid[16475], build[default/rpm/537cb22/2020-05-28T14:47:19.882936Z], OS[Linux/3.10.0-862.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2020-06-24T16:19:54,881][INFO ][o.e.n.Node               ] [8.1.10.146] JVM arguments [-Xms31g, -Xmx31g, -Xmn16g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss5m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-8074107225992765619, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm]
[2020-06-24T16:19:56,533][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [aggs-matrix-stats]
[2020-06-24T16:19:56,533][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [analysis-common]
[2020-06-24T16:19:56,534][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [ingest-common]
[2020-06-24T16:19:56,534][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [ingest-geoip]
[2020-06-24T16:19:56,534][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [ingest-user-agent]
[2020-06-24T16:19:56,534][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [lang-expression]
[2020-06-24T16:19:56,534][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [lang-mustache]
[2020-06-24T16:19:56,534][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [lang-painless]
[2020-06-24T16:19:56,534][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [mapper-extras]
[2020-06-24T16:19:56,534][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [parent-join]
[2020-06-24T16:19:56,534][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [percolator]
[2020-06-24T16:19:56,534][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [rank-eval]
[2020-06-24T16:19:56,534][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [reindex]
[2020-06-24T16:19:56,534][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [repository-url]
[2020-06-24T16:19:56,534][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [transport-netty4]
[2020-06-24T16:19:56,534][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [tribe]
[2020-06-24T16:19:56,534][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [x-pack-ccr]
[2020-06-24T16:19:56,534][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [x-pack-core]
[2020-06-24T16:19:56,534][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [x-pack-deprecation]
[2020-06-24T16:19:56,534][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [x-pack-graph]
[2020-06-24T16:19:56,535][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [x-pack-ilm]
[2020-06-24T16:19:56,535][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [x-pack-logstash]
[2020-06-24T16:19:56,535][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [x-pack-ml]
[2020-06-24T16:19:56,535][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [x-pack-monitoring]
[2020-06-24T16:19:56,535][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [x-pack-rollup]
[2020-06-24T16:19:56,535][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [x-pack-security]
[2020-06-24T16:19:56,535][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [x-pack-sql]
[2020-06-24T16:19:56,535][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [x-pack-upgrade]
[2020-06-24T16:19:56,535][INFO ][o.e.p.PluginsService     ] [8.1.10.146] loaded module [x-pack-watcher]
[2020-06-24T16:19:56,535][INFO ][o.e.p.PluginsService     ] [8.1.10.146] no plugins loaded
[2020-06-24T16:19:59,636][INFO ][o.e.x.s.a.s.FileRolesStore] [8.1.10.146] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2020-06-24T16:20:00,145][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [8.1.10.146] [controller/16672] [Main.cc@109] controller (64 bit): Version 6.8.10 (Build a5f7163bca0250) Copyright (c) 2020 Elasticsearch BV
[2020-06-24T16:20:00,539][DEBUG][o.e.a.ActionModule       ] [8.1.10.146] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2020-06-24T16:20:00,720][INFO ][o.e.d.DiscoveryModule    ] [8.1.10.146] using discovery type [zen] and host providers [settings]
[2020-06-24T16:20:01,839][INFO ][o.e.n.Node               ] [8.1.10.146] initialized
[2020-06-24T16:20:01,839][INFO ][o.e.n.Node               ] [8.1.10.146] starting ...
[2020-06-24T16:20:01,961][INFO ][o.e.t.TransportService   ] [8.1.10.146] publish_address {8.1.10.146:9300}, bound_addresses {8.1.10.146:9300}
[2020-06-24T16:20:01,971][INFO ][o.e.b.BootstrapChecks    ] [8.1.10.146] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-06-24T16:20:01,974][ERROR][o.e.b.Bootstrap          ] [8.1.10.146] node validation exception
[1] bootstrap checks failed
[1]: memory locking requested for elasticsearch process but memory is not locked
[2020-06-24T16:20:01,978][INFO ][o.e.n.Node               ] [8.1.10.146] stopping ...
[2020-06-24T16:20:02,078][INFO ][o.e.n.Node               ] [8.1.10.146] stopped
[2020-06-24T16:20:02,078][INFO ][o.e.n.Node               ] [8.1.10.146] closing ...
[2020-06-24T16:20:02,087][INFO ][o.e.n.Node               ] [8.1.10.146] closed
[2020-06-24T16:20:02,089][INFO ][o.e.x.m.p.NativeController] [8.1.10.146] Native controller process has stopped - no new native processes can be started

limits.conf 配置如下

[root@iimhisdbs2 elasticsearch]# 

cat /etc/security/limits.conf     

*                  soft    nofile      65536
*                  hard    nofile      65536
*                  soft    nproc       65536
*                  hard    nproc       65536
# End of file
* soft memlock unlimited
* hard memlock unlimited



[root@iimhisdbs2 elasticsearch]# ulimit -H -l  #先执行一下这条命令  看是否生效
unlimited


我的解决办法

##
vim /usr/lib/systemd/system/elasticsearch.service
[Service]
RuntimeDirectory=elasticsearch
PrivateTmp=true
Environment=ES_HOME=/usr/share/elasticsearch
Environment=ES_PATH_CONF=/etc/elasticsearch
Environment=PID_DIR=/var/run/elasticsearch
EnvironmentFile=-/etc/sysconfig/elasticsearch
LimitMEMLOCK=infinity  #添加这行

https://www.cnblogs.com/zhuzi91/p/8228214.html
https://stackoverflow.com/questions/51382869/unable-to-lock-jvm-memory-error-12-reason-cannot-allocate-memory-elasticsear

 

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!