How to manage two or more consumers via pthreads?

Question

I have a generic problem I am looking to solve, where chunks of binary data sent from a standard input or regular file stream to an application, which in turn converts that bina

Answer 1

Ah. So I think I misunderstood the question.... sorry.

I had thought you wanted to run gunzip and then one other internal filter, and wanted to do that 'N' times.

It seems what you really want to do is run many stages of filters, one after the other... some using external commands and some (perhaps ?) internal to the program. Hence the desire to manage some inter-stage buffering.

So... I've had another go at this. The objective is to run any number of stages, starting with the input stage, then extrenal command or internal function "filter" stages, and finally the output stage. Each external command stage had three pthreads -- for stdin, stdout and stderr. Internal function stages use one pthread and the initial input and final output one pthread each. Between the stages is a small pipe structure (called a "straw") to "double buffer" and decouple the stages... I hope this is closer to what you had in mind.

The "straw" is the essence of the thing:

struct straw
{
  pthread_mutex_t*  mutex ;

  struct lump*  free ;
  pthread_cond_t* free_cond ;
  bool          free_waiting ;

  struct lump*  ready ;
  pthread_cond_t* ready_cond ;
  bool          ready_waiting ;

  struct lump*  lumps[2] ;
} ;

where a struct lump contains a buffer and what-not. The "straw" has two such "lumps", and at any moment one pthread may be filling one lump, while another is draining the other. Or both lumps may be free (on the free list) or both ready (full) waiting on the ready list.

Then to aquire an empty lump to fill it (eg when reading from a pipe):

static struct lump*
lump_acquire(struct straw* strw)
{
  struct lump* lmp ;

  pthread_mutex_lock(strw->mutex) ;

  while (strw->free == NULL)
    {
      strw->free_waiting = true ;
      pthread_cond_wait(strw->free_cond, strw->mutex) ;
      strw->free_waiting = false ;
    } ;

  lmp = strw->free ;
  strw->free = lmp->next ;

  pthread_mutex_unlock(strw->mutex) ;

  lmp->next = NULL ;                    /* tidy                 */

  lmp->ptr  = lmp->end = lmp->buff ;    /* empty                */
  lmp->done = false ;

  return lmp ;
} ;

Then to blow the completed lump into (one end of) the straw.

static void
lump_blow(struct lump* lmp)
{
  struct straw* strw ;

  strw = lmp->strw ;

  qassert((lmp == strw->lumps[0]) || (lmp == strw->lumps[1])) ;
  qassert( (lmp->buff <= lmp->ptr)
                     && (lmp->ptr <= lmp->end)
                                 && (lmp->end <= lmp->limit) ) ;

  lmp->ptr = lmp->buff ;

  pthread_mutex_lock(strw->mutex) ;

  if (strw->ready == NULL)
    strw->ready = lmp ;
  else
    strw->ready->next = lmp ;
  lmp->next = NULL ;

  if (strw->ready_waiting)
    pthread_cond_signal(strw->ready_cond) ;

  pthread_mutex_unlock(strw->mutex) ;
} ;

To suck a lump out of (the other end of) the straw:

static struct lump*
lump_suck(struct straw* strw)
{
  struct lump* lmp ;

  pthread_mutex_lock(strw->mutex) ;

  while (strw->ready == NULL)
    {
      strw->ready_waiting = true ;
      pthread_cond_wait(strw->ready_cond, strw->mutex) ;
      strw->ready_waiting = false ;
    } ;

  lmp = strw->ready ;
  strw->ready = lmp->next ;

  pthread_mutex_unlock(strw->mutex) ;

  qassert( (lmp->buff <= lmp->ptr)
                     && (lmp->ptr <= lmp->end)
                                 && (lmp->end <= lmp->limit) ) ;

  lmp->ptr = lmp->buff ;        /* lmp->ptr..lmp->end   */
  lmp->next = NULL ;            /* tidy                 */

  return lmp ;
} ;

And the final piece, freeing a lump once it has been drained:

static void
lump_free(struct lump* lmp)
{
  struct straw* strw ;

  strw = lmp->strw ;

  qassert((lmp == strw->lumps[0]) || (lmp == strw->lumps[1])) ;
  qassert( (lmp->buff <= lmp->ptr)
                     && (lmp->ptr <= lmp->end)
                                 && (lmp->end <= lmp->limit) ) ;

  pthread_mutex_lock(strw->mutex) ;

  if (strw->free == NULL)
    strw->free = lmp ;
  else
    strw->free->next = lmp ;

  lmp->next = NULL ;                    /* end of list of free  */

  lmp->ptr  = lmp->end = lmp->buff ;    /* empty                */
  lmp->done = false ;

  if (strw->free_waiting)
    pthread_cond_signal(strw->free_cond) ;

  pthread_mutex_unlock(strw->mutex) ;
} ;

The entire program is too big to fit in an answer -- see: pipework.c where that starts:

/*==============================================================================
 * pipework.c
 *
 * Copyright (c) Chris Hall (GMCH) 2014, All rights reserved.
 *
 * Though you may do what you like with this, provided you recognise that
 * it is offered "as is", gratis, and may or may not be fit for any purpose
 * whatsoever -- you are on your own.
 *
 *------------------------------------------------------------------------------
 *
 * This will read from stdin, pass the data through an arbitrary number of
 * "filter" stages and finally write the result to stdout.
 *
 * A filter stage may be an external command taking a piped stdin and
 * outputting to a piped stdout.  Anything it says to stderr is collected
 * and output to the program's stderr.
 *
 * A filter stage may also be an internal function.
 *
 * The input, filter and output stages are implemented as a number of pthreads,
 * with internal, miniature pipes (called "straws") between them.  All I/O is
 * blocking.  This is an experiment in the use of pthreads to simplify things.
 *
 * ============================
 * This is v0.08 of  4-Jul-2014
 * ============================
 *
 * The 'main' below runs eight stages: input, 4 commands, 2 internal filters
 * and the output.  The output should be an exact copy of the input.
 *
 * In order to test the stderr handling, the following small perl script is
 * used as two of the command filters:
 *
 *   chatter.pl
 *   --------------------------------------------------------
     use strict ;
     use warnings ;

     my $line = 0 ;
     while ()
       {
         my $len = length($_) ;
         my $r   = rand ;

         $line += 1 ;
         print STDERR "|$line:$len:$r|" ;

         if (int($r * 100) == 0)
           {
             print STDERR "\n" ;
           } ;

         print $_ ;
       } ;
 *   --------------------------------------------------------
 *
 *------------------------------------------------------------------------------
 * Limitations
 *
 *   * this will crash if it gets some error its not expecting or not
 *     designed to overcome.  Clearly, to be useful this needs to be more
 *     robust and more informative.
 *
 *   * some (possible/theoretical) errors are simply ignored.
 *
 *   * no attempt is made to collect up the child processes or to discover
 *     their return codes.  If the child process reports errors or anything
 *     else on stderr, then that will be visible.  But otherwise, if it just
 *     crashes then the pipeline will run to completion, but the result may
 *     be nonsense.
 *
 *   * if one of the child processes stalls, the whole thing stalls.
 *
 *   * an I/O error in a stage will send 'end' downstream, but the program
 *     will continue until everything upstream has completed.
 *
 *   * generally... not intended for production use !!
 */

And the perl script is available as: chatter.pl

HTH