How to buffer stdout in memory and write it from a dedicated thread

Question

I have a C application with many worker threads. It is essential that these do not block so where the worker threads need to write to a file on disk, I have them write to a

Answer 1

You can change how buffering works with setvbuf() or setbuf(). There's a description here: http://publications.gbdirect.co.uk/c_book/chapter9/input_and_output.html.

[Edit]

stdout really is a FILE*. If the existing code works with FILE*s, I don't see what prevents it from working with stdout.

Answer 2

If you're working with the GNU libc, you might use memory streams.

Answer 3

Why don't you wrap your entire application in another? Basically, what you want is a smart cat that copies stdin to stdout, buffering as necessary. Then use standard stdin/stdout redirection. This can be done without modifying your current application at all.

~MSalters/# YourCurrentApp | bufcat

Answer 4

The method using the 4096 bigbuf will only sort of work. I've tried this code, and while it does successfully capture stdout into the buffer, it's unusable in a real world case. You have no way of knowing how long the captured output is, so no way of knowing when to terminate the string '\0'. If you try to use the buffer you get 4000 characters of garbage spit out if you had successfully captured 96 characters of stdout output.

In my application, I'm using a perl interpreter in the C program. I have no idea how much output is going to be spit out of what ever document is thrown at the C program, and hence the code above would never allow me to cleanly print that output out anywhere.

Answer 5

You can "redirect" stdout into file using freopen().

man freopen says:

The freopen() function opens the file whose name is the string pointed to by path and associates the stream pointed to by stream with it. The original stream (if it exists) is closed. The mode argument is used just as in the fopen() function. The primary use of the freopen() function is to change the file associated with a standard text stream (stderr, stdin, or stdout).

This file well could be a pipe - worker threads will write to that pipe and writer thread will listen.

Answer 6

I like the idea of using freopen. You might also be able to redirect stdout to a pipe using dup and dup2, and then use read to grab data from the pipe.

Something like so:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

#define MAX_LEN 40

int main( int argc, char *argv[] ) {
  char buffer[MAX_LEN+1] = {0};
  int out_pipe[2];
  int saved_stdout;

  saved_stdout = dup(STDOUT_FILENO);  /* save stdout for display later */

  if( pipe(out_pipe) != 0 ) {          /* make a pipe */
    exit(1);
  }

  dup2(out_pipe[1], STDOUT_FILENO);   /* redirect stdout to the pipe */
  close(out_pipe[1]);

  /* anything sent to printf should now go down the pipe */
  printf("ceci n'est pas une pipe");
  fflush(stdout);

  read(out_pipe[0], buffer, MAX_LEN); /* read from pipe into buffer */

  dup2(saved_stdout, STDOUT_FILENO);  /* reconnect stdout for testing */
  printf("read: %s\n", buffer);

  return 0;
}