How can I write blocking in stdout with node.js?

Question

I\'m writing a node.js application which stdout is piped to a file. I\'m writing everything with console.log. After a while my Application reaches the 1GB Limit and stops. T

Answer 1

The problem (exploding memory usage) probably occurs because your program is creating output faster than it can be display. Therefore you want to throttle it. Your question requests "synchronous output", but actually the problem can be solved by using purely "asynchronous"(*) code.

(* NOTE: In this post the term "asynchronous" is used in the "javascript-single-thread" sense. That differs from the conventional "multi-thread" sense which an entirely different kettle of fish).

This answer shows how "asynchronous" code can be used with Promises to prevent memory usage from exploding by "pausing" (as opposed to blocking) execution until the write output has been successfully flushed. This answer also explains how an asynchronous code solution can be advantageous compared to a synchronous code solution.

Q: "Paused" sounds like "blocked", and how can asynchronous code possibly "block"? That's an oxymoron!

A: It works because the the javascript v8 engine pauses (blocks) execution of only the single code slice await an asynchronous promise to complete, while permitting other code slices to execute in the meantime.

Here is an asynchronous write function (adapted from here).

async function streamWriteAsync(
  stream,
  chunk,
  encoding='utf8') {
  return await new Promise((resolve, reject) => {
    const errListener = (err) => {
      stream.removeListener('error', errListener);
      reject(err);
    };
    stream.addListener('error', errListener);
    const callback = () => {
      stream.removeListener('error', errListener);
      resolve(undefined);
    };
    stream.write(chunk, encoding, callback);
  });
}

It can be called from an asynchrous function in your source code, e.g.
case 1

async function main() {
  while (true)
    await streamWriteAsync(process.stdout, 'hello world\n')
}
main();

Where main() is the only function called from the top level. The memory usage will not explode as it would if calling console.log('hello world');.

More context is required to clearly see the advantage over a true synchronous write:
case 2

async function logger() {
  while (true)
    await streamWriteAsync(process.stdout, 'hello world\n')
}
const snooze = ms => new Promise(resolve => setTimeout(resolve, ms));
function allowOtherThreadsToRun(){
  return Promise(resolve => setTimeout(resolve, 0));
}
async function essentialWorker(){
  let a=0,b=1;
  while (true) {
    let tmp=a; a=b; b=tmp;
    allowOtherThreadsToRun();
  }
}
async function main(){
  Promise.all([logger(), essentialWorker()])  
}
main();

Running the above code (case 2) would show that the memory use is still not exploding (same as case 1) because the logger associated slice was paused, but the CPU usage was still because as the essentialWorker slice was not paused - which is good (think COVID).

In comparison, a synchronous solution would also block the essentialWorker.

What happens with multiple slices calling streamWrite?
case 3

async function loggerHi() {
  while (true)
    await streamWriteAsync(process.stdout, 'hello world\n')
}
async function loggerBye() {
  while (true)
    await streamWriteAsync(process.stdout, 'goodbye world\n')
}
function allowOtherThreadsToRun(){
  return Promise(resolve => setTimeout(resolve, 0));
}
async function essentialWorker(){
  let a=0,b=1;
  while (true) {
    let tmp=a; a=b; b=tmp;
    allowOtherThreadsToRun();
  }
}
async function main(){
  Promise.all([loggerHi(), loggerBye(), essentialWorker()])  
}
main();

In this case (case 3), the memory usage is bound, and the essentialWorker CPU usage is high, the same as in case 2. Individual lines of hello world and goodbye world would remain atomic, but the lines would not alternate cleanly, e.g.,

...
hello world 
hello world 
goodbye world 
hello world 
hello world 
...

could appear.

Answer 2

Edit: as indicated by commenter, this solution has a problem. printResolver could be made into an array, but using top solution is much easier

A synchronous print function that also works with pipes aka FIFO's, using Async/await. Make sure you always call "print" with "await print"

let printResolver;
process.stdout.on('drain', function () {
    if (printResolver) printResolver();
});

async function print(str) {
    var done = process.stdout.write(str);
    if (!done) {
        await new Promise(function (resolve) {
            printResolver = resolve;
        });
    }
}

Answer 3

Write using process.stdout.write, the return value is whether data got buffered. If it's true, continue writing when process.stdout emits the drain event.

If you want your code to look sync, use streamlinejs as described here: Node.js stdout flush

Answer 4

Don't.

What you want to do is pause() your input when the output is full, like the pump() method does, then resume() it when you've got space to write. If not, your process balloons out to gargantuan size.

You probably want to use the more direct outputStream stuff for that, though, or the write() call, not console.log().

Answer 5

If you really really want synchronous writes to stdout you can do:

var fs = require('fs');
fs.writeSync(1, "Foo\n");
fs.fsyncSync(1);