Java example of using ExecutorService and PipedReader/PipedWriter (or PipedInputStream/PipedOutputStream) for consumer-producer

Question

I'm looking for a simple producer - consumer implementation in Java and don't want to reinvent the wheel

I couldn't find an example that uses both the new concurrency package and either of the Piped classes

Is there an example for using both PipedInputStream and the new Java concurrency package for this?

Is there a better way without using the Piped classes for such a task?

Answer 1

For your task it might be sufficient to just use a single thread and write to the file using a BufferedOutputStream as you are reading from the database.

If you want more control over the buffer size and the size of chunks being written to the file, you can do something like this:

class Producer implements Runnable {

    private final OutputStream out;
    private final SomeDBClass db;

    public Producer( OutputStream out, SomeDBClass db ){
        this.out = out;
        this.db = db;
    }

    public void run(){
        // If you're writing to a text file you might want to wrap
        // out in a Writer instead of using `write` directly.
        while( db has more data ){
            out.write( the data );
        }
        out.flush();
        out.close();
    }
}

class Consumer implements Runnable {

    private final InputStream in;
    private final OutputStream out;
    public static final int CHUNKSIZE=512;

    public Consumer( InputStream in, OutputStream out ){
        this.out = out;
        this.in = in;
    }

    public void run(){
        byte[] chunk = new byte[CHUNKSIZE];

        for( int bytesRead; -1 != (bytesRead = in.read(chunk,0,CHUNKSIZE) );;){
            out.write(chunk, 0, bytesRead);
        }
        out.close();
    }
}

And in the calling code:

FileOutputStream toFile = // Open the stream to a file
SomeDBClass db = // Set up the db connection
PipedInputStream pi = new PipedInputStream(); // Optionally specify a size
PipedOutputStream po = new PipedOutputStream( pi );

ExecutorService exec = Executors.newFixedThreadPool(2);
exec.submit( new Producer( po, db ) );
exec.submit( new Consumer( pi, toFile ) );
exec.shutdown();

• Also catch any exceptions that might be thrown.

Note that if this is all you're doing, there isn't any advantage to using an ExecutorService. Executors are useful when you have many tasks (too many to launch all of them in threads at the same time). Here you have only two threads that have to run at the same time, so calling Thread#start directly will have less overhead.