JSON.stringify large object optimization

问题

I'd like to convert a large JSON object to string using the JSON.stringify, but due to the size of the object I got an error of

<--- Last few GCs --->

[20817:0x2cc2830]   295727 ms: Scavenge 1335.8 (1423.9) -> 1335.6 (1427.9) MB, 7.7 / 0.0 ms  (average mu = 0.255, current mu = 0.170) allocation failure 
[20817:0x2cc2830]   295966 ms: Mark-sweep 1339.5 (1427.9) -> 1339.3 (1422.9) MB, 227.1 / 0.0 ms  (average mu = 0.272, current mu = 0.291) allocation failure scavenge might not succeed
[20817:0x2cc2830]   295990 ms: Scavenge 1343.2 (1422.9) -> 1343.0 (1426.9) MB, 9.2 / 0.0 ms  (average mu = 0.272, current mu = 0.291) allocation failure 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x16f9b389e549 <JSObject>
    0: builtin exit frame: stringify(this=0x16f9b38877f9 <Object map = 0x39fb09082ba1>,0x1da31d6826f1 <undefined>,0x2762ba9be041 <JSFunction replacer (sfi = 0x1bc744b67dc9)>,0x1deddcf2e701 <Object map = 0x6bcf8da0341>,0x16f9b38877f9 <Object map = 0x39fb09082ba1>)

    1: stringify [0x16c685254241] [/home/blingga/Documents/integrations/integration-flow17employment-hrplus-to-datalake/dist/flow17em...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8c02c0 node::Abort() [node]
 2: 0x8c030c  [node]
 3: 0xad15de v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xad1814 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xebe752  [node]
 6: 0xebe858 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [node]
 7: 0xeca982 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xecb2b4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xecdf21 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
10: 0xe962a5  [node]
11: 0xe9dc0b v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [node]
12: 0x11cd6b0 v8::internal::IncrementalStringBuilder::Extend() [node]
13: 0xf92c20 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
14: 0xf9719d v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
15: 0xf977ee v8::internal::JsonStringifier::SerializeArrayLikeSlow(v8::internal::Handle<v8::internal::JSReceiver>, unsigned int, unsigned int) [node]
16: 0xf93e96 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
17: 0xf9719d v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
18: 0xf977ee v8::internal::JsonStringifier::SerializeArrayLikeSlow(v8::internal::Handle<v8::internal::JSReceiver>, unsigned int, unsigned int) [node]
19: 0xf93e96 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
20: 0xf9719d v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
21: 0xf97ef9 v8::internal::JsonStringifier::Stringify(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [node]
22: 0xba0051 v8::internal::Builtin_JsonStringify(int, v8::internal::Object**, v8::internal::Isolate*) [node]
23: 0x2639339dc17d 
Aborted (core dumped)

Is there any way I can optimize the JSON.stringify? Or is there anyway I can convert my JSON to string? Thanks.

回答1:

did you try big-json?

const fs = require('fs');
const path = require('path');
const json = require('big-json');

const readStream = fs.createReadStream('big.json');
const parseStream = json.createParseStream();

parseStream.on('data', function(pojo) {
    // => receive reconstructed POJO
});

readStream.pipe(parseStream);

回答2:

You may also want to try JSONStream. It let's you stringify or parse a JSON virtually of any size.

Here's a parse example from their docs:

var request = require('request')
, JSONStream = require('JSONStream')
, es = require('event-stream')

request({url: 'http://isaacs.couchone.com/registry/_all_docs'})
.pipe(JSONStream.parse('rows.*'))
.pipe(es.mapSync(function (data) {
  console.error(data)
  return data
}))

来源：https://stackoverflow.com/questions/55468777/json-stringify-large-object-optimization

标签

node.js

json

string

amazon-s3

aws-sdk