问题
I have a cluster of cassandra (v3.11.4) with 14 nodes and I wanna to add a new node. The machine has 256GB memory and I set heap size (max and min) to 64GB . But I cannot add a new node due to memory error! What is the exact problem and What I need to do?
The last line of logs are as follows:
INFO [CompactionExecutor:1153] 2020-06-11 10:51:09,719 NoSpamLogger.java:91 - Maximum memory usage reached (20971520000), cannot allocate chunk of 1048576
INFO [Service Thread] 2020-06-11 10:51:10,339 GCInspector.java:284 - G1 Young Generation GC in 239ms. G1 Eden Space: 3254779904 -> 0; G1 Old Gen: 27209478712 -> 25011094408;
INFO [Service Thread] 2020-06-11 10:51:11,217 GCInspector.java:284 - G1 Young Generation GC in 456ms. G1 Eden Space: 3254779904 -> 0; G1 Old Gen: 25011094408 -> 23790284880; G1 Survivor Space: 167772160 -> 268435456;
INFO [Service Thread] 2020-06-11 10:51:12,192 GCInspector.java:284 - G1 Young Generation GC in 486ms. G1 Eden Space: 3154116608 -> 0; G1 Old Gen: 23790284880 -> 23936030504; G1 Survivor Space: 268435456 -> 301989888;
vmop [threads: total initially_running wait_to_block] [time: spin block sync cleanup vmop] page_trap_count
5829.747: G1IncCollectionPause [ 772 18 22 ] [ 0 0 1 1 118 ] 0
....
5872.925: G1IncCollectionPause [ 774 17 21 ] [ 0 0 0 1 172 ] 0
vmop [threads: total initially_running wait_to_block] [time: spin block sync cleanup vmop] page_trap_count
5876.138: G1IncCollectionPause [ 774 16 24 ] [ 0 0 1 1 148 ] 0
....
6075.988: G1IncCollectionPause [ 772 22 23 ] [ 0 0 1 1 170 ] 0
vmop [threads: total initially_running wait_to_block] [time: spin block sync cleanup vmop] page_trap_count
6079.968: G1IncCollectionPause [ 772 0 0 ] [ 0 0 1405 1 135 ] 0
6084.827: G1IncCollectionPause [ 772 0 1 ] [ 0 0 88 1 218 ] 0
.....
6218.542: no vm operation [ 773 13 21 ] [ 0 0 1 1 0 ] 0
6219.252: G1IncCollectionPause [ 773 19 19 ] [ 0 0 0 1 158 ] 0
6219.751: G1CollectFull [ 773 11 20 ] [ 0 1 1 1 75023 ] 0
WARN [Service Thread] 2020-06-11 10:52:32,407 GCInspector.java:282 - G1 Old Generation GC in 67243ms. G1 Eden Space: 2919235584 -> 0; G1 Old Gen: 24289926888 -> 19483266320; G1 Survivor Space: 671088640 -> 0;
INFO [Service Thread] 2020-06-11 10:52:32,408 StatusLogger.java:47 - Pool Name Active Pending Completed Blocked All Time Blocked
WARN [GossipTasks:1] 2020-06-11 10:52:32,506 Gossiper.java:783 - Gossip stage has 9 pending tasks; skipping status check (no nodes will be marked down)
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e3160f000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
INFO [Service Thread] 2020-06-11 10:52:32,515 StatusLogger.java:51 - MutationStage 576 257 5688240 0 0
INFO [Service Thread] 2020-06-11 10:52:32,521 StatusLogger.java:51 - ViewMutationStage 0 0 0 0 0
INFO [Service Thread] 2020-06-11 10:52:32,521 StatusLogger.java:51 - ReadStage 0 0 0 0 0
INFO [Service Thread] 2020-06-11 10:52:32,522 StatusLogger.java:51 - RequestResponseStage 0 0 13 0 0
INFO [Service Thread] 2020-06-11 10:52:32,522 StatusLogger.java:51 - ReadRepairStage 0 0 0 0 0
INFO [Service Thread] 2020-06-11 10:52:32,523 StatusLogger.java:51 - CounterMutationStage 0 0 0 0 0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8d172f6000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140243956360960 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,523 StatusLogger.java:51 - MiscStage 0 0 0 0 0
INFO [Service Thread] 2020-06-11 10:52:32,524 StatusLogger.java:51 - CompactionExecutor 33 33 1068 0 0
INFO [Service Thread] 2020-06-11 10:52:32,524 StatusLogger.java:51 - MemtableReclaimMemory 0 0 221 0 0
INFO [Service Thread] 2020-06-11 10:52:32,525 StatusLogger.java:51 - PendingRangeCalculator 0 0 16 0 0
INFO [Service Thread] 2020-06-11 10:52:32,525 StatusLogger.java:51 - GossipStage 0 0 18945 0 0
INFO [Service Thread] 2020-06-11 10:52:32,525 StatusLogger.java:51 - SecondaryIndexManagement 0 0 0 0 0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31652000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248691058432 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,526 StatusLogger.java:51 - HintsDispatcher 0 0 0 0 0
Java HotSpot(TM) 64-Bit Server VM warning: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31561000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
INFO: os::commit_memory(0x00007f8e3269d000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)[thread 140248690071296 also had an error]
[thread 140248708142848 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31b35000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248696182528 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,526 StatusLogger.java:51 - MigrationStage 0 0 27 0 0
INFO [Service Thread] 2020-06-11 10:52:32,526 StatusLogger.java:51 - MemtablePostFlush 1 1 304 0 0
INFO [Service Thread] 2020-06-11 10:52:32,526 StatusLogger.java:51 - PerDiskMemtableFlushWriter_0 1 1 222 0 0
INFO [Service Thread] 2020-06-11 10:52:32,527 StatusLogger.java:51 - ValidationExecutor 0 0 0 0 0
INFO [Service Thread] 2020-06-11 10:52:32,527 StatusLogger.java:51 - Sampler 0 0 0 0 0
INFO [Service Thread] 2020-06-11 10:52:32,527 StatusLogger.java:51 - MemtableFlushWriter 1 1 184 0 0
INFO [Service Thread] 2020-06-11 10:52:32,528 StatusLogger.java:51 - InternalResponseStage 0 0 25 0 0
INFO [Service Thread] 2020-06-11 10:52:32,528 StatusLogger.java:51 - AntiEntropyStage 0 0 0 0 0
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
INFO [Service Thread] 2020-06-11 10:52:32,528 StatusLogger.java:51 - CacheCleanupExecutor 0 0 0 0 0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e327a1000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248709207808 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.INFO: os::commit_memory(0x00007f8e31af4000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248695916288 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e316de000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248691631872 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,530 StatusLogger.java:61 - CompactionManager 33 35
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:73 - MessagingService n/a 0/0
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:83 - Cache Type Size Capacity KeysToSave
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:85 - KeyCache 3211792520 10485760000 all
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:91 - RowCache 0 0 all
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:98 - Table Memtable ops,data
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:101 - system_distributed.parent_repair_history 0,0
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8ceedd1000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140243279877888 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:101 - system_distributed.repair_history 60,5088
INFO [Service Thread] 2020-06-11 10:52:32,531 StatusLogger.java:101 - system_distributed.view_build_status 0,0
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.compaction_history 182,47068
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.schema_aggregates 0,0
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.schema_triggers 0,0
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.size_estimates 0,0
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.paxos 0,0
# An error report file with more information is saved as:
# /home/dev/hs_err_pid11578.log
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.views_builds_in_progress 0,0
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.batches 0,0
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.schema_keyspaces 0,0
INFO [Service Thread] 2020-06-11 10:52:32,532 StatusLogger.java:101 - system.sstable_activity 6713,123062
INFO [Service Thread] 2020-06-11 10:52:32,533 StatusLogger.java:101 - system.batchlog 0,0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e100fb000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248131811072 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e3150a000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
INFO [Service Thread] 2020-06-11 10:52:32,533 StatusLogger.java:101 - system.schema_columns 0,0
[thread 140248689714944 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,534 StatusLogger.java:101 - system.hints 0,0
INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.IndexInfo 0,0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f7caaf9e000, 262144, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248944940800 also had an error]INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.schema_columnfamilies 0,0
INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.schema_functions 0,0
INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.built_views 0,0
INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.peer_events 0,0
INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.range_xfers 0,0
INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.peers 0,0
INFO [Service Thread] 2020-06-11 10:52:32,535 StatusLogger.java:101 - system.transferred_ranges 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system.schema_usertypes 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system.local 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system.available_ranges 4,20360
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system.prepared_statements 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system_schema.columns 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system_schema.types 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system_schema.indexes 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system_schema.keyspaces 0,0
INFO [Service Thread] 2020-06-11 10:52:32,536 StatusLogger.java:101 - system_schema.dropped_columns 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - system_schema.aggregates 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - system_schema.triggers 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - system_schema.tables 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - system_schema.views 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - system_schema.functions 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxxxxxxxts 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxxxxxet 0,0
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxxxage 7883,699041
INFO [Service Thread] 2020-06-11 10:52:32,537 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxxs 4824,213056
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxlist 705216,33023249
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxxxxxxxxs 357,256470
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - sxxxxxxxxxxxxxxxxe 0,0
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxxxxmembers 6482,260169
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - sxxxxxxxxxxxxid 0,0
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxcount 84961,1532062
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxe 25736,10620210
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31762000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248692172544 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxxes 5511,4021957
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxxs 0,0
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxd 0,0
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxxe 1460,1127389
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxxs 468,131122
INFO [Service Thread] 2020-06-11 10:52:32,538 StatusLogger.java:101 - xxxxxxxd_users 2313247,95919246
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxxs 76,36133
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxx.users 2287211,201973918
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e34df8000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248749410048 also had an error]
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxxt 16,1040
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxx 0,0
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxxe 0,0
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxo 72693,16357850
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - xxxxxd 26143,1492343
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_auth.roles 0,0
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_auth.role_members 0,0
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_auth.resource_role_permissons_index 0,0
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_auth.role_permissions 0,0
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_traces.sessions 0,0
INFO [Service Thread] 2020-06-11 10:52:32,539 StatusLogger.java:101 - system_traces.events 0,0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31693000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248691324672 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e32599000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248707077888 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e34e39000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248749676288 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e3a07e000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
INFO: os::commit_memory(0x00007f8e315af000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)[thread 140248835942144 also had an error]
[thread 140248690390784 also had an error]Java HotSpot(TM) 64-Bit Server VM warning:
Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8430b9f000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140205730166528 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e3a72e000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248842954496 also had an error]Java HotSpot(TM) 64-Bit Server VM warning:
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
INFO: os::commit_memory(0x00007f8e11541000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248153069312 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f7803456000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140153427945216 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8b5f2c9000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140236574201600 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e318a9000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248693511936 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8d15901000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140243929143040 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e318ea000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248693778176 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31868000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248693245696 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31b79000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248696461056 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8e31d40000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140248698324736 also had an error]
[thread 140216167200512 also had an error][thread 140248204584704 also had an error][thread 140245662537472 also had an error][thread 140209602766592 also had an error][thread 140230211868416 also had an error]
[thread 140248151602944 also had an error][thread 140210503980800 also had an error][thread 140230032652032 also had an error][thread 140198818285312 also had an error][thread 140248755021568 also had an error][thread 140248765159168 also had an error][thread 140248224515840 also had an error][thread 140230234380032 also had an error][thread 140243956045568 also had an error][thread 140248768407296 also had an error][thread 140248294921984 also had an error][thread 140248768141056 also had an error][thread 140207186040576 also had an error]
[thread 140245662004992 also had an error][thread 140248585135872 also had an error]
The amount of available memory is as follow:
I appreciate any help!
I Also see these warning in start of the joining:
INFO [main] 2020-06-11 09:08:21,598 CassandraDaemon.java:493 - JVM Arguments: [-Xloggc:./apache-cassandra-3.11.4/bin/../logs/gc.log, -ea, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+UseNUMA, -XX:+PerfDisableSharedMem, -Djava.net.preferIPv4Stack=true, -Xms64G, -Xmx64G, -XX:+UseG1GC, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -XX:+PrintAdaptiveSizePolicy, -XX:+PrintGCTimeStamps, -XX:+PrintSafepointStatistics, -XX:+PrintClassHistogramBeforeFullGC, -XX:+PrintClassHistogramAfterFullGC, -XX:CompileCommandFile=./apache-cassandra-3.11.4/bin/../conf/hotspot_compiler, -javaagent:./apache-cassandra-3.11.4/bin/../lib/jamm-0.3.0.jar, -Dcassandra.jmx.remote.port=7199, -Dcom.sun.management.jmxremote.rmi.port=7199, -Dcom.sun.management.jmxremote.authenticate=true, -Dcom.sun.management.jmxremote.password.file=/home/dev/apache-cassandra-3.11.4/conf/jmxremote.password, -Dcom.sun.management.jmxremote.access.file=/home/dev/apache-cassandra-3.11.4/conf/jmxremote.access, -Djava.library.path=./apache-cassandra-3.11.4/bin/../lib/sigar-bin, -Djava.rmi.server.hostname=172.17.41.172, -XX:OnOutOfMemoryError=kill -9 %p, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=./apache-cassandra-3.11.4/bin/../logs, -Dcassandra.storagedir=./apache-cassandra-3.11.4/bin/../data]
WARN [main] 2020-06-11 09:08:21,659 NativeLibrary.java:187 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
WARN [main] 2020-06-11 09:08:21,660 StartupChecks.java:136 - jemalloc shared library could not be preloaded to speed up memory allocations
INFO [main] 2020-06-11 09:08:21,661 StartupChecks.java:176 - JMX is enabled to receive remote connections on port: 7199
INFO [main] 2020-06-11 09:08:21,663 SigarLibrary.java:44 - Initializing SIGAR library
INFO [main] 2020-06-11 09:08:21,677 SigarLibrary.java:180 - Checked OS settings and found them configured for optimal performance.
WARN [main] 2020-06-11 09:08:21,681 StartupChecks.java:311 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl.
回答1:
The error message says that the JVM failed to commit memory, because mmap syscall returned with the error code 12 (ENOMEM).
This typically happens when the process reaches one of the OS memory limits:
The number of memory mappings exceeded vm.max_map_count sysctl limit. This is a quite common problem with Cassandra, since Cassandra tends to mmap thousands files, and the default limit is rather low - around 65K.
How to check:
wc -l /proc/<pid>/mapswhere<pid>is Java process ID.How to fix:
sudo sysctl vm.max_map_count=1000000Total amount of the process' virtual memory exceeded RLIMIT_AS
How to check:
ulimit -v,cat /proc/<pid>/status | grep VmHow to fix:
ulimit -v unlimitedbefore starting Cassandra in the same shell.Overcommit is disabled, and the overcommit ratio is too low.
How to check:
sysctl vm.overcommit_memory,sysctl vm.overcommit_ratio,cat /proc/meminfoHow to fix:
sudo sysctl vm.overcommit_memory=0
来源：https://stackoverflow.com/questions/62356714/cassandra-add-node-problem-failed-error-cannot-allocate-memory-errno-12