fs.createWriteStream over several processes

问题

How can I implement a system where multiple Node.js processes write to the same file with fs.createWriteStream, such that they don't overwrite data? It looks like the default setup for fs.createWriteStream is that the file is cleared out when that method is called. My goal is to clear out the file once, and then have all other subsequent writers only append data.

Should I use fs.createWriteStream and then fs.appendFile? Or is there a way to open up a stream for each process, not just for the first process to open the file?

回答1:

Should I use fs.createWriteStream and then fs.appendFile?

you can use either.

with fs.createWriteStream you have to change the flag like this:

fs.createWriteStream('your_file',{
  flags: 'a+', // default is 'w' (just 'a' might be enough here, i'm not sure)
})

this should create the file if it doesn't exist or open it with write access if it exists and set the pointer to end. (append mode)

How to use fs.appendFile should be clear and it does pretty much the same.

Now the problem with multiple processes accessing the same file. Obviously only one process can open the same file with write access at the same time. Therefore you need to wait for the file to be released if another process has the write access. You will probably need a library for that.

this one for example: https://www.npmjs.com/package/lockup
or this one: https://github.com/Perennials/mutex-node
you can also find alot more here: https://www.npmjs.com/browse/keyword/lock
or here: https://www.npmjs.com/browse/keyword/mutex

I have not tried any of those libraries but the one I posted and several others on the list should do exactly what you need.

回答2:

Writing on a single file from multiple processes, ensuring data integrity, it is a fairly complex operation that you can orchestrate using File locking.

However, you have two simpler approaches:

1. Writing on a temporary file for each process, and then concatenate the files at the end of the operations.
2. Transmitting what you need to write to a dedicated, single process and delegate the writing execution to it. Keep in mind that sending messages among processes can be expensive.

来源：https://stackoverflow.com/questions/36658995/fs-createwritestream-over-several-processes