How to manage two or more consumers via pthreads?

Question

I have a generic problem I am looking to solve, where chunks of binary data sent from a standard input or regular file stream to an application, which in turn converts that bina

Answer 1

Regarding the part 'How to manage two or more consumers via pthreads?' of your post let me cite these points about 'Designing Threaded Programs':

In general though, in order for a program to take advantage of Pthreads, it must be able to be organized into discrete, independent tasks which can execute concurrently. For example, if routine1 and routine2 can be interchanged, interleaved and/or overlapped in real time, they are candidates for threading.

and

Several common models for threaded programs exist:

• Manager/worker: a single thread, the manager assigns work to other threads, the workers. Typically, the manager handles all input and
  parcels out work to the other tasks. At least two forms of the
  manager/worker model are common: static worker pool and dynamic
  worker pool.
• Pipeline: a task is broken into a series of suboperations, each of which is handled in series, but concurrently, by a different thread.
  An automobile assembly line best describes this model.
• Peer: similar to the manager/worker model, but after the main thread creates other threads, it participates in the work.

Regarding your problem...

The problem I am facing is that I need to add a sleep(1) statement in consume_gunzip_chunk, before doing the read, in order to get things working properly.

Eric Lippert Best Practices with Multithreading in C# might not solve it but, they should help you finding the right solution to your multi-threaded program, particularly points 5, and 8:

5.At all costs avoid shared memory. Most threading bugs are caused by a failure to understand real-world shared memory semantics. If you must make threads, treat them as though they were processes: give them everything they need to do their work and let them work without modifying the memory associated with any other thread. Just like a process doesn't get to modify the memory of any other process.

8.If you use Thread.Sleep with an argument other than zero or one in any production code, you are possibly doing something wrong. Threads are expensive; you don't pay a worker to sleep, so don't pay a thread to sleep either. If you are using sleeps to solve a correctness issue by avoiding a timing problem -- as you appear to be in your code -- then you definitely have done something deeply wrong. Multithreaded code needs to be correct irrespective of accidents of timing.

Answer 2

I will start by saying that I feel pthreads conditions and mutexes were not really necessary here, nor was non-blocking I/O the best reaction to the problems you describe.

In my opinion, the problems you describe with your condition- and mutex-less version are symptoms of forgetting to close() assiduously the ends of your pipes, with the result that a copy of the writing-end file descriptor of the pipe feeding the child process's stdin leaked (into that child or others) alive.

Then, given that a writing-end corresponding to stdin's reading-end still existed, the system did not give EOF but instead blocked indefinitely.

In your case, you did prevent the pipe-end file descriptors from leaking to the spawned child (with the correct close() calls on the child-side of the fork() within your popen3(), although you forgot to close() the wrong-end pipe ends on the parent-side). However, you did not prevent this leakage to all other children! If you call popen3() twice, the leakage of the set of three descriptors into the child is prevented, but as the parent still owns them, when the next call to popen3() happens, after the fork() there are now 6 file descriptors to close (The old set of three and and the new set of three you just created).

In your case, therefore, you should set the close-on-exec flag on those pipe ends, thusly:

fcntl(fdIn [PIPEWR], F_SETFD, fcntl(fdIn [PIPEWR], F_GETFD) | FD_CLOEXEC);
fcntl(fdOut[PIPERD], F_SETFD, fcntl(fdOut[PIPERD], F_GETFD) | FD_CLOEXEC);
fcntl(fdErr[PIPERD], F_SETFD, fcntl(fdErr[PIPERD], F_GETFD) | FD_CLOEXEC);

Here is code that spawns 6 threads and 3 processes, and passes its input unmodified to the output, after internally compressing then decompressing it. It effectively implements gzip -c - | XOR 0x55 | XOR 0x55 | gunzip -c - | cat, where:

1. Standard input is fed to to gzip by thread srcThrd.
2. gzip's output is read by thread a2xor0Thrd and fed to thread xor0Thrd.
3. Thread xor0Thrd XORs its input with 0x55 before passing it on to thread xor1Thrd.
4. Thread xor1Thrd XORs its input with 0x55 before passing it on to thread xor22BThrd.
5. Thread xor22BThrd feeds its input to process gunzip.
6. Process gunzip feeds its output directly (without going through a thread) to cat
7. Process cat's output is read by thread dstThrd and printed to standard output.

Compression is done by inter-process pipe communication, while XORing is done by intra-process pipe communication. No mutexes or condition variables are used. main() is extremely easy to understand. This code should be easy to extend to your situation.

/* Includes */
#include <stdlib.h>
#include <pthread.h>
#include <unistd.h>
#include <stdio.h>
#include <fcntl.h>



/* Defines */
#define PIPERD 0
#define PIPEWR 1




/* Data structures */
typedef struct PIPESET{
    int Ain[2];
    int Aout[2];
    int Aerr[2];
    int xor0[2];
    int xor1[2];
    int xor2[2];
    int Bin[2];
    int BoutCin[2];
    int Berr[2];
    int Cout[2];
    int Cerr[2];
} PIPESET;




/* Function Implementations */

/**
 * Source thread main method.
 * 
 * Slurps from standard input and feeds process A.
 */

void* srcThrdMain(void* arg){
    PIPESET* pipeset = (PIPESET*)arg;

    char c;
    while(read(0, &c, 1) > 0){
        write(pipeset->Ain[PIPEWR], &c, 1);
    }

    close(pipeset->Ain[PIPEWR]);

    pthread_exit(NULL);
}

/**
 * A to XOR0 thread main method.
 * 
 * Manually pipes from standard output of process A to input of thread XOR0.
 */

void* a2xor0ThrdMain(void* arg){
    PIPESET* pipeset = (PIPESET*)arg;

    char    buf[65536];
    ssize_t bytesRead;

    while((bytesRead = read(pipeset->Aout[PIPERD], buf, 65536)) > 0){
        write(pipeset->xor0[PIPEWR], buf, bytesRead);
    }

    close(pipeset->xor0[PIPEWR]);

    pthread_exit(NULL);
}

/**
 * XOR0 thread main method.
 * 
 * XORs input with 0x55 and outputs to input of XOR1.
 */

void* xor0ThrdMain(void* arg){
    PIPESET* pipeset = (PIPESET*)arg;

    char c;
    while(read(pipeset->xor0[PIPERD], &c, 1) > 0){
        c ^= 0x55;
        write(pipeset->xor1[PIPEWR], &c, 1);
    }

    close(pipeset->xor1[PIPEWR]);

    pthread_exit(NULL);
}

/**
 * XOR1 thread main method.
 * 
 * XORs input with 0x55 and outputs to input of process B.
 */

void* xor1ThrdMain(void* arg){
    PIPESET* pipeset = (PIPESET*)arg;

    char c;
    while(read(pipeset->xor1[PIPERD], &c, 1) > 0){
        c ^= 0x55;
        write(pipeset->xor2[PIPEWR], &c, 1);
    }

    close(pipeset->xor2[PIPEWR]);

    pthread_exit(NULL);
}

/**
 * XOR2 to B thread main method.
 * 
 * Manually pipes from input (output of XOR1) to input of process B.
 */

void* xor22BThrdMain(void* arg){
    PIPESET* pipeset = (PIPESET*)arg;

    char    buf[65536];
    ssize_t bytesRead;

    while((bytesRead = read(pipeset->xor2[PIPERD], buf, 65536)) > 0){
        write(pipeset->Bin[PIPEWR], buf, bytesRead);
    }

    close(pipeset->Bin[PIPEWR]);

    pthread_exit(NULL);
}

/**
 * Destination thread main method.
 * 
 * Manually copies the standard output of process C to the standard output.
 */

void* dstThrdMain(void* arg){
    PIPESET* pipeset = (PIPESET*)arg;

    char c;
    while(read(pipeset->Cout[PIPERD], &c, 1) > 0){
        write(1, &c, 1);
    }

    pthread_exit(NULL);
}

/**
 * Set close on exec flag on given descriptor.
 */

void setCloExecFlag(int fd){
    fcntl(fd, F_SETFD, fcntl(fd, F_GETFD) | FD_CLOEXEC);
}

/**
 * Set close on exec flag on given descriptor.
 */

void unsetCloExecFlag(int fd){
    fcntl(fd, F_SETFD, fcntl(fd, F_GETFD) & ~FD_CLOEXEC);
}

/**
 * Pipe4.
 * 
 * Create a pipe with some ends possibly marked close-on-exec.
 */

#define PIPE4_FLAG_NONE       (0U)
#define PIPE4_FLAG_RD_CLOEXEC (1U << 0)
#define PIPE4_FLAG_WR_CLOEXEC (1U << 1)

int pipe4(int fd[2], int flags){
    int ret = pipe(fd);

    if(flags&PIPE4_FLAG_RD_CLOEXEC){setCloExecFlag(fd[PIPERD]);}
    if(flags&PIPE4_FLAG_WR_CLOEXEC){setCloExecFlag(fd[PIPEWR]);}

    return ret;
}

/**
 * Pipe4 explicit derivatives.
 */

#define pipe4_cloexec(fd)  pipe4((fd), PIPE4_FLAG_RD_CLOEXEC|PIPE4_FLAG_WR_CLOEXEC)

/**
 * Popen4.
 * 
 * General-case for spawning a process and tethering it with cloexec pipes on stdin,
 * stdout and stderr.
 * 
 * @param [in]      cmd    The command to execute.
 * @param [in/out]  pin    The pointer to the cloexec pipe for stdin.
 * @param [in/out]  pout   The pointer to the cloexec pipe for stdout.
 * @param [in/out]  perr   The pointer to the cloexec pipe for stderr.
 * @param [in]      flags  A bitwise OR of flags to this function. Available
 *                         flags are:
 * 
 *     POPEN4_FLAG_NONE:
 *         Explicitly specify no flags.
 *     POPEN4_FLAG_NOCLOSE_PARENT_STDIN,
 *     POPEN4_FLAG_NOCLOSE_PARENT_STDOUT,
 *     POPEN4_FLAG_NOCLOSE_PARENT_STDERR:
 *         Don't close pin[PIPERD], pout[PIPEWR] and perr[PIPEWR] in the parent,
 *         respectively.
 *     POPEN4_FLAG_CLOSE_CHILD_STDIN,
 *     POPEN4_FLAG_CLOSE_CHILD_STDOUT,
 *     POPEN4_FLAG_CLOSE_CHILD_STDERR:
 *         Close the respective streams in the child. Ignores pin, pout and perr
 *         entirely. Overrides a NOCLOSE_PARENT flag for the same stream.
 */

#define POPEN4_FLAG_NONE                             (0U)
#define POPEN4_FLAG_NOCLOSE_PARENT_STDIN        (1U << 0)
#define POPEN4_FLAG_NOCLOSE_PARENT_STDOUT       (1U << 1)
#define POPEN4_FLAG_NOCLOSE_PARENT_STDERR       (1U << 2)
#define POPEN4_FLAG_CLOSE_CHILD_STDIN           (1U << 3)
#define POPEN4_FLAG_CLOSE_CHILD_STDOUT          (1U << 4)
#define POPEN4_FLAG_CLOSE_CHILD_STDERR          (1U << 5)

pid_t popen4(const char* cmd, int pin[2], int pout[2], int perr[2], int flags){
    /********************
     **  FORK PROCESS  **
     ********************/
    pid_t ret = fork();

    if(ret < 0){
        /**
         * Error in fork(), still in parent.
         */

        fprintf(stderr, "fork() failed!\n");
        return ret;
    }else if(ret == 0){
        /**
         * Child-side of fork
         */

        if(flags & POPEN4_FLAG_CLOSE_CHILD_STDIN){
            close(0);
        }else{
            unsetCloExecFlag(pin [PIPERD]);
            dup2(pin [PIPERD], 0);
        }
        if(flags & POPEN4_FLAG_CLOSE_CHILD_STDOUT){
            close(1);
        }else{
            unsetCloExecFlag(pout[PIPEWR]);
            dup2(pout[PIPEWR], 1);
        }
        if(flags & POPEN4_FLAG_CLOSE_CHILD_STDERR){
            close(2);
        }else{
            unsetCloExecFlag(perr[PIPEWR]);
            dup2(perr[PIPEWR], 2);
        }

        execl("/bin/sh", "sh", "-c", cmd, NULL);

        fprintf(stderr, "exec() failed!\n");
        exit(-1);
    }else{
        /**
         * Parent-side of fork
         */

        if(~flags & POPEN4_FLAG_NOCLOSE_PARENT_STDIN  &&
           ~flags & POPEN4_FLAG_CLOSE_CHILD_STDIN){
            close(pin [PIPERD]);
        }
        if(~flags & POPEN4_FLAG_NOCLOSE_PARENT_STDOUT &&
           ~flags & POPEN4_FLAG_CLOSE_CHILD_STDOUT){
            close(pout[PIPEWR]);
        }
        if(~flags & POPEN4_FLAG_NOCLOSE_PARENT_STDERR &&
           ~flags & POPEN4_FLAG_CLOSE_CHILD_STDERR){
            close(perr[PIPEWR]);
        }

        return ret;
    }

    /* Unreachable */
    return ret;
}

/**
 * Main Function.
 * 
 * Sets up the whole piping scheme.
 */

int main(int argc, char* argv[]){
    pthread_t srcThrd, a2xor0Thrd, xor0Thrd, xor1Thrd, xor22BThrd, dstThrd;
    pid_t     gzip, gunzip, cat;
    PIPESET   pipeset;

    pipe4_cloexec(pipeset.Ain);
    pipe4_cloexec(pipeset.Aout);
    pipe4_cloexec(pipeset.Aerr);
    pipe4_cloexec(pipeset.Bin);
    pipe4_cloexec(pipeset.BoutCin);
    pipe4_cloexec(pipeset.Berr);
    pipe4_cloexec(pipeset.Cout);
    pipe4_cloexec(pipeset.Cerr);
    pipe4_cloexec(pipeset.xor0);
    pipe4_cloexec(pipeset.xor1);
    pipe4_cloexec(pipeset.xor2);

    /* Spawn processes */
    gzip   = popen4("gzip -c -",   pipeset.Ain,     pipeset.Aout,    pipeset.Aerr, POPEN4_FLAG_NONE);
    gunzip = popen4("gunzip -c -", pipeset.Bin,     pipeset.BoutCin, pipeset.Berr, POPEN4_FLAG_NONE);
    cat    = popen4("cat",         pipeset.BoutCin, pipeset.Cout,    pipeset.Cerr, POPEN4_FLAG_NONE);


    /* Spawn threads */
    pthread_create(&srcThrd,    NULL, srcThrdMain,    &pipeset);
    pthread_create(&a2xor0Thrd, NULL, a2xor0ThrdMain, &pipeset);
    pthread_create(&xor0Thrd,   NULL, xor0ThrdMain,   &pipeset);
    pthread_create(&xor1Thrd,   NULL, xor1ThrdMain,   &pipeset);
    pthread_create(&xor22BThrd, NULL, xor22BThrdMain, &pipeset);
    pthread_create(&dstThrd,    NULL, dstThrdMain,    &pipeset);
    pthread_join(srcThrd,    (void**)NULL);
    pthread_join(a2xor0Thrd, (void**)NULL);
    pthread_join(xor0Thrd,   (void**)NULL);
    pthread_join(xor1Thrd,   (void**)NULL);
    pthread_join(xor22BThrd, (void**)NULL);
    pthread_join(dstThrd,    (void**)NULL);
    return 0;
}

Commentary on your own code

There are many issues with your code, most of which have nothing to do with threading.

• You don't close() the file descriptor d->gunzip_ptr->in. This means that gunzip can never know that no more input is forthcoming on its stdin, so it will never exit.
• Since gunzip doesn't ever exit, it will never close() its stdout, and thus a blocking read() at the other end will never unblock. A non-blocking read will instead always give -1, with errno == EAGAIN.
• Your popen3() doesn't close() p_stdin[POPEN3_READ], p_stdout[POPEN3_WRITE] or p_stderr[POPEN3_WRITE] on the parent side of the fork(). Only the child should have those descriptors. Failing to close these means that when the parent itself tries to read the stdout and stderr of the child, it will never see EOF, again for the same reasons as above: Because it itself still owns a write-end pipe in which it could write, making new data appear to the read-end.
• Your code implicitly relies on gunzip writing out at least one byte for every 1024 you write in. There is no guarantee that this will be the case, since gunzip may, at its leisure, buffer internally.
  • This is because your code reads then copies chunks of at most BUF_LENGTH_VALUE bytes into d->in_buf. You then assign the number of bytes you read through fread() to d->n_in_bytes. This same d->n_in_bytes is used in your write() call to write to gunzip's stdin. You then signal for consume_gunzip_chunk() to wake up, then pthread_cond_wait()'s for the next gzip-compressed chunk. But this gzip-compressed chunk may never come, since there is no guarantee that gunzip will be able to unpack useful output from just the first 1024 bytes of input, nor even a guarantee that it will write() it out instead of buffering it until it has, say, 4096 bytes (a full page) of output. Therefore, the read() call in consume_gunzip_chunk() may never succeed (or even return, if read() was blocking). And if read() never returns, then consume_gunzip_chunk() doesn't signal d->in_cond, and so all three threads get stuck. And even if read() is non-blocking, the last block of output from gzip may never come, since gzip's input is never closed, so it doesn't flush out its buffers by write()'ing them out, so read() on the other end will never get useful data and no amount of pleading will elicit it without a close().

• POSSIBLE (LIKELY?) CAUSE OF BUG: d->n_out_bytes_read_from_gunzip, once it becomes non-0, will never become 0 again. This means that the extremely baffling

  while (d->n_in_bytes != 0 || d->n_out_bytes_read_from_gunzip != 0)
      pthread_cond_wait(&d->in_cond, &d->in_lock);
  

  within produce_gzip_chunk() will, once entered with d->n_out_bytes_read_from_gunzip != 0, forever remain stuck. By calling sleep(1) within consume_gunzip_chunk(), which sets d->n_out_bytes_read_from_gunzip, you may have defused the problem by reading all input before consume_gunzip_chunk() could lock up the system by setting d->n_out_bytes_read_from_gunzip to a non-zero value.

• There are two threads that call pthread_cond_wait(&d->in_cond, &d->in_lock);, these being produce_gzip_chunk() and consume_gzip_chunk(). There is absolutely no guarantee that when consume_gunzip_chunk() calls pthread_cond_signal(&d->in_cond);, that the "correct" thread (whichever it is in your design) will receive the signal. To ensure that all of them will, use pthread_cond_broadcast(), but then you expose yourself to the thundering herd problem. Needing to use pthread_cond_broadcast() in this situation is, again, a symptom of a bad design in my opinion.
• Related, you call pthread_cond_signal(&d->in_cond) within a thread (indeed, a function) in which you call pthread_cond_wait(&d->in_cond, &d->in_lock). What purpose does that serve?
• You use d->in_lock for too many disparate purposes, exposing yourself to the possibility of deadlock, or low performance due to excessive protection. In particular you use it as the protection for both d->in_cond and d->out_cond. This is too strong a protection – the output of gunzip into d->in_line should be able to happen simultaneously with the input of gunzip being written into and out of d->in_buf.

• Within consume_gunzip_chunk(), you have

  while (d->n_in_bytes_written_to_gunzip == 0) {
      pthread_cond_wait(&d->out_cond, &d->in_lock);
  }
  if (d->n_in_bytes_written_to_gunzip) {
      ...
  

  This if can never fail! Is there a case you may have in mind?

• Consider making the entire struct pthread_data volatile (or at least those integer elements which are used by multiple threads), since the compiler might decide to optimize out loads and stores that should, in fact, remain.

Compliments

So as to not sound too negative, I would like to say that in general your problems were not due to misuse of the pthreads API but due to erroneous consumer-producer logic and lack of close()s. Additionally, you appear to understand that pthread_cond_wait() may wake up spuriously, and so you have wrapped it up in a loop that checks the invariant.

In the future:

I would use pipes, even between threads. This absolves you from needing to implement your own consumer-producer scheme; The kernel has solved it for you already, and provides you with the pipe(), read() and write() primitives, which are all you need to take advantage of this ready-made solution. It also makes the code cleaner and void of mutexes and condition variables. One must simply be diligent in closing the ends, and one must be supremely careful around pipes in the presence of fork(). The rules are simple:

• If a pipe's write-end exists, a read() on an open read-end will not give EOF but will block or EAGAIN.
• If a pipe's write-ends have all been closed, a read() on an open read-end will give EOF.
• If a pipe's read-ends have all been closed, a write() to any of its write-ends will cause SIGPIPE.
• fork() duplicates the entire process, including all descriptors (modulo maybe crazy stuff in pthread_atfork())!

Answer 3

Ah. So I think I misunderstood the question.... sorry.

I had thought you wanted to run gunzip and then one other internal filter, and wanted to do that 'N' times.

It seems what you really want to do is run many stages of filters, one after the other... some using external commands and some (perhaps ?) internal to the program. Hence the desire to manage some inter-stage buffering.

So... I've had another go at this. The objective is to run any number of stages, starting with the input stage, then extrenal command or internal function "filter" stages, and finally the output stage. Each external command stage had three pthreads -- for stdin, stdout and stderr. Internal function stages use one pthread and the initial input and final output one pthread each. Between the stages is a small pipe structure (called a "straw") to "double buffer" and decouple the stages... I hope this is closer to what you had in mind.

The "straw" is the essence of the thing:

struct straw
{
  pthread_mutex_t*  mutex ;

  struct lump*  free ;
  pthread_cond_t* free_cond ;
  bool          free_waiting ;

  struct lump*  ready ;
  pthread_cond_t* ready_cond ;
  bool          ready_waiting ;

  struct lump*  lumps[2] ;
} ;

where a struct lump contains a buffer and what-not. The "straw" has two such "lumps", and at any moment one pthread may be filling one lump, while another is draining the other. Or both lumps may be free (on the free list) or both ready (full) waiting on the ready list.

Then to aquire an empty lump to fill it (eg when reading from a pipe):

static struct lump*
lump_acquire(struct straw* strw)
{
  struct lump* lmp ;

  pthread_mutex_lock(strw->mutex) ;

  while (strw->free == NULL)
    {
      strw->free_waiting = true ;
      pthread_cond_wait(strw->free_cond, strw->mutex) ;
      strw->free_waiting = false ;
    } ;

  lmp = strw->free ;
  strw->free = lmp->next ;

  pthread_mutex_unlock(strw->mutex) ;

  lmp->next = NULL ;                    /* tidy                 */

  lmp->ptr  = lmp->end = lmp->buff ;    /* empty                */
  lmp->done = false ;

  return lmp ;
} ;

Then to blow the completed lump into (one end of) the straw.

static void
lump_blow(struct lump* lmp)
{
  struct straw* strw ;

  strw = lmp->strw ;

  qassert((lmp == strw->lumps[0]) || (lmp == strw->lumps[1])) ;
  qassert( (lmp->buff <= lmp->ptr)
                     && (lmp->ptr <= lmp->end)
                                 && (lmp->end <= lmp->limit) ) ;

  lmp->ptr = lmp->buff ;

  pthread_mutex_lock(strw->mutex) ;

  if (strw->ready == NULL)
    strw->ready = lmp ;
  else
    strw->ready->next = lmp ;
  lmp->next = NULL ;

  if (strw->ready_waiting)
    pthread_cond_signal(strw->ready_cond) ;

  pthread_mutex_unlock(strw->mutex) ;
} ;

To suck a lump out of (the other end of) the straw:

static struct lump*
lump_suck(struct straw* strw)
{
  struct lump* lmp ;

  pthread_mutex_lock(strw->mutex) ;

  while (strw->ready == NULL)
    {
      strw->ready_waiting = true ;
      pthread_cond_wait(strw->ready_cond, strw->mutex) ;
      strw->ready_waiting = false ;
    } ;

  lmp = strw->ready ;
  strw->ready = lmp->next ;

  pthread_mutex_unlock(strw->mutex) ;

  qassert( (lmp->buff <= lmp->ptr)
                     && (lmp->ptr <= lmp->end)
                                 && (lmp->end <= lmp->limit) ) ;

  lmp->ptr = lmp->buff ;        /* lmp->ptr..lmp->end   */
  lmp->next = NULL ;            /* tidy                 */

  return lmp ;
} ;

And the final piece, freeing a lump once it has been drained:

static void
lump_free(struct lump* lmp)
{
  struct straw* strw ;

  strw = lmp->strw ;

  qassert((lmp == strw->lumps[0]) || (lmp == strw->lumps[1])) ;
  qassert( (lmp->buff <= lmp->ptr)
                     && (lmp->ptr <= lmp->end)
                                 && (lmp->end <= lmp->limit) ) ;

  pthread_mutex_lock(strw->mutex) ;

  if (strw->free == NULL)
    strw->free = lmp ;
  else
    strw->free->next = lmp ;

  lmp->next = NULL ;                    /* end of list of free  */

  lmp->ptr  = lmp->end = lmp->buff ;    /* empty                */
  lmp->done = false ;

  if (strw->free_waiting)
    pthread_cond_signal(strw->free_cond) ;

  pthread_mutex_unlock(strw->mutex) ;
} ;

The entire program is too big to fit in an answer -- see: pipework.c where that starts:

/*==============================================================================
 * pipework.c
 *
 * Copyright (c) Chris Hall (GMCH) 2014, All rights reserved.
 *
 * Though you may do what you like with this, provided you recognise that
 * it is offered "as is", gratis, and may or may not be fit for any purpose
 * whatsoever -- you are on your own.
 *
 *------------------------------------------------------------------------------
 *
 * This will read from stdin, pass the data through an arbitrary number of
 * "filter" stages and finally write the result to stdout.
 *
 * A filter stage may be an external command taking a piped stdin and
 * outputting to a piped stdout.  Anything it says to stderr is collected
 * and output to the program's stderr.
 *
 * A filter stage may also be an internal function.
 *
 * The input, filter and output stages are implemented as a number of pthreads,
 * with internal, miniature pipes (called "straws") between them.  All I/O is
 * blocking.  This is an experiment in the use of pthreads to simplify things.
 *
 * ============================
 * This is v0.08 of  4-Jul-2014
 * ============================
 *
 * The 'main' below runs eight stages: input, 4 commands, 2 internal filters
 * and the output.  The output should be an exact copy of the input.
 *
 * In order to test the stderr handling, the following small perl script is
 * used as two of the command filters:
 *
 *   chatter.pl
 *   --------------------------------------------------------
     use strict ;
     use warnings ;

     my $line = 0 ;
     while (<STDIN>)
       {
         my $len = length($_) ;
         my $r   = rand ;

         $line += 1 ;
         print STDERR "|$line:$len:$r|" ;

         if (int($r * 100) == 0)
           {
             print STDERR "\n" ;
           } ;

         print $_ ;
       } ;
 *   --------------------------------------------------------
 *
 *------------------------------------------------------------------------------
 * Limitations
 *
 *   * this will crash if it gets some error its not expecting or not
 *     designed to overcome.  Clearly, to be useful this needs to be more
 *     robust and more informative.
 *
 *   * some (possible/theoretical) errors are simply ignored.
 *
 *   * no attempt is made to collect up the child processes or to discover
 *     their return codes.  If the child process reports errors or anything
 *     else on stderr, then that will be visible.  But otherwise, if it just
 *     crashes then the pipeline will run to completion, but the result may
 *     be nonsense.
 *
 *   * if one of the child processes stalls, the whole thing stalls.
 *
 *   * an I/O error in a stage will send 'end' downstream, but the program
 *     will continue until everything upstream has completed.
 *
 *   * generally... not intended for production use !!
 */

And the perl script is available as: chatter.pl

HTH