Is it safe to pipe the output of several parallel processes to one file using >>?

Question

I\'m scraping data from the web, and I have several processes of my scraper running in parallel.

I want the output of each of these processes to end up in the same f

Answer 1

Briefly, no. >> doesn't respect multiple processes.

Answer 2

You'll need to ensure that you're writing whole lines in single write operations (so if you're using some form of stdio, you'll need to set it for line buffering for at least the length of the longest line that you can output.) Since the shell uses O_APPEND for the >> redirection then all your writes will then automatically append to the file with no further action on your part.

Answer 3

Generally, no.

On Linux this might be possible, as long as two conditions are met: each line is written in a one operation, and the line is no longer than PIPE_SIZE (usually the same as PAGE_SIZE, usually 4096). But... I wouldn't count on that; this behaviour might change.

It is better to use some kind of real logging mechanism, like syslog.