问题
Here's the error message:
<--- Last few GCs --->
[2383:0x7efe08001450] 6100 ms: Scavenge 30.3 (39.5) -> 30.5 (42.7) MB, 73.5 / 0.0 ms (average mu = 1.000, curr
ent mu = 1.000) allocation failure
[2383:0x7efe08001450] 8464 ms: Scavenge 35.1 (44.5) -> 35.3 (44.8) MB, 2336.2 / 0.0 ms (average mu = 1.000, cu
rrent mu = 1.000) allocation failure
[2383:0x7efe08001450] 32349 ms: Scavenge 36.1 (44.8) -> 36.0 (45.8) MB, 23879.5 / 0.2 ms (average mu = 1.000, c
urrent mu = 1.000) allocation failure
<--- JS stacktrace --->
FATAL ERROR: Scavenger: semi-space copy Allocation failed - JavaScript heap out of memory
1: 0xa2b020 node::Abort() [node]
2: 0x97a467 node::FatalError(char const*, char const*) [node]
3: 0xb9e0ee v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xb9e467 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xd3e875 [node]
6: 0xdab25e [node]
7: 0xdae060 v8::internal::Scavenger::Process(v8::internal::OneshotBarrier*) [node]
8: 0xdaeb29 v8::internal::ScavengingTask::ProcessItems() [node]
9: 0xdaecd1 v8::internal::ScavengingTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner) [node]
10: 0xd590f9 v8::internal::ItemParallelJob::Run() [node]
11: 0xdaf9c5 v8::internal::ScavengerCollector::CollectGarbage() [node]
12: 0xd37aba v8::internal::Heap::Scavenge() [node]
13: 0xd4d37b v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [nod
e]
14: 0xd4de65 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReaso
n, v8::GCCallbackFlags) [node]
15: 0xd5082c v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal
::AllocationOrigin, v8::internal::AllocationAlignment) [node]
16: 0xd20e82 v8::internal::Factory::AllocateRawWithAllocationSite(v8::internal::Handle<v8::internal::Map>, v8::inte
rnal::AllocationType, v8::internal::Handle<v8::internal::AllocationSite>) [node]
17: 0xd20f38 v8::internal::Factory::NewJSObjectFromMap(v8::internal::Handle<v8::internal::Map>, v8::internal::Alloc
ationType, v8::internal::Handle<v8::internal::AllocationSite>) [node]
18: 0xd2130e v8::internal::Factory::NewJSArrayWithUnverifiedElements(v8::internal::Handle<v8::internal::FixedArrayB
ase>, v8::internal::ElementsKind, int, v8::internal::AllocationType) [node]
19: 0xd24a32 v8::internal::Factory::NewJSArray(v8::internal::ElementsKind, int, int, v8::internal::ArrayStorageAllo
cationMode, v8::internal::AllocationType) [node]
20: 0xf6edbb v8::internal::ValueDeserializer::ReadDenseJSArray() [node]
21: 0xf6f1cd v8::internal::ValueDeserializer::ReadObjectInternal() [node]
22: 0xf6f338 v8::internal::ValueDeserializer::ReadObject() [node]
23: 0xf6d80d v8::internal::ValueDeserializer::ReadJSObjectProperties(v8::internal::Handle<v8::internal::JSObject>,
v8::internal::SerializationTag, bool) [node]
24: 0xf6ec06 v8::internal::ValueDeserializer::ReadSparseJSArray() [node]
25: 0xf6f118 v8::internal::ValueDeserializer::ReadObjectInternal() [node]
26: 0xf6f338 v8::internal::ValueDeserializer::ReadObject() [node]
27: 0xf6e186 v8::internal::ValueDeserializer::ReadJSObjectProperties(v8::internal::Handle<v8::internal::JSObject>,
v8::internal::SerializationTag, bool) [node]
28: 0xf6e959 v8::internal::ValueDeserializer::ReadJSObject() [node]
29: 0xf6f140 v8::internal::ValueDeserializer::ReadObjectInternal() [node]
30: 0xf6f338 v8::internal::ValueDeserializer::ReadObject() [node]
31: 0xf6d80d v8::internal::ValueDeserializer::ReadJSObjectProperties(v8::internal::Handle<v8::internal::JSObject>,
v8::internal::SerializationTag, bool) [node]
32: 0xf6e959 v8::internal::ValueDeserializer::ReadJSObject() [node]
33: 0xf6f140 v8::internal::ValueDeserializer::ReadObjectInternal() [node]
34: 0xf6f338 v8::internal::ValueDeserializer::ReadObject() [node]
35: 0xba6e15 v8::ValueDeserializer::ReadValue(v8::Local<v8::Context>) [node]
36: 0xa720ff node::worker::Message::Deserialize(node::Environment*, v8::Local<v8::Context>) [node]
37: 0xa72d86 node::worker::MessagePort::ReceiveMessage(v8::Local<v8::Context>, bool) [node]
38: 0xa72ee4 node::worker::MessagePort::OnMessage() [node]
39: 0x13247e9 [node]
40: 0x1336c50 [node]
41: 0x13250f8 uv_run [node]
42: 0xae3533 node::worker::Worker::Run() [node]
43: 0xae3fc8 [node]
44: 0x7eff53bdafa3 [/lib/x86_64-linux-gnu/libpthread.so.0]
45: 0x7eff53b0b4cf clone [/lib/x86_64-linux-gnu/libc.so.6]
node server on GCP - Linux x64, 96 cores, 135GB RAM. I'm processing an array with 100.000.000 items, divided into 95 cores with creating workers.
This is just one of the error types I get, I got NewArray Allocation failed as well for example. I was playing around with number of workers, --max-old-space-size, --max-semi-space-size, but none works. The most interesting thing is my mac with 6 cores, 16GB ram (only 10 given to node) runs well, just slow, with increasing the hardware resources, I'm expecting faster results.
This is the first time I'm trying to process this amount of data, and multi-threading with node, what am I doing wrong?
来源：https://stackoverflow.com/questions/62469659/scavenger-allocation-failed-javascript-heap-out-of-memory