Shutdown “worker” go routine after buffer is empty

Question

I want my go routine worker (ProcessToDo() in the code below) to wait until all \"queued\" work is processed before shutting down.

The worker routine

Answer 1

It's usually a bad idea to have a consumer of a channel close it, since sending on a closed channel is a panic.

In this case, if you never want to interrupt the consumer before all messages have been sent, just use a for...range loop and close the channel when you're done. You will also need a signal like a WaitGroup to wait for the goroutine to finish (rather than using time.Sleep)

http://play.golang.org/p/r97vRPsxEb

var wg sync.WaitGroup

func ProcessToDo(todo chan string) {
    defer wg.Done()
    for work := range todo {
        fmt.Printf("todo: %q\n", work)
        time.Sleep(100 * time.Millisecond)

    }
    fmt.Printf("Shutting down ProcessToDo - todo channel closed!\n")

}

func main() {
    todo := make(chan string, 100)
    wg.Add(1)
    go ProcessToDo(todo)

    for i := 0; i < 20; i++ {
        todo <- fmt.Sprintf("Message %02d", i)
    }

    fmt.Println("*** all messages queued ***")
    close(todo)
    wg.Wait()
}

Answer 2

done channel in your case is completely unnecessary as you can signal the shutdown by closing the todo channel itself.

And use the for range on the channel which will iterate until the channel is closed and its buffer is empty.

You should have a done channel, but only so that the goroutine itself can signal that it finished work and so the main goroutine can continue or exit.

This variant is equivalent to yours, is much simpler and does not require time.Sleep() calls to wait other goroutines (which would be too erroneous and undeterministic anyway). Try it on the Go Playground:

func ProcessToDo(done chan struct{}, todo chan string) {
    for work := range todo {
        fmt.Printf("todo: %q\n", work)
        time.Sleep(100 * time.Millisecond)
    }
    fmt.Printf("Shutting down ProcessToDo - todo channel closed!\n")
    done <- struct{}{} // Signal that we processed all jobs
}

func main() {
    done := make(chan struct{})
    todo := make(chan string, 100)

    go ProcessToDo(done, todo)

    for i := 0; i < 20; i++ {
        todo <- fmt.Sprintf("Message %02d", i)
    }

    fmt.Println("*** all messages queued ***")
    close(todo)
    <-done // Wait until the other goroutine finishes all jobs
}

Also note that worker goroutines should signal completion using defer so the main goroutine won't get stuck waiting for the worker if it returns in some unexpected way, or panics. So it should rather start like this:

defer func() {
    done <- struct{}{} // Signal that we processed all jobs
}()

You can also use sync.WaitGroup to sync the main goroutine to the worker (to wait it up). In fact if you plan to use multiple worker goroutines, that is cleaner than to read multiple values from the done channel. Also it's simpler to signal the completion with WaitGroup as it has a Done() method (which is a function call) so you don't need an anonymous function:

defer wg.Done()

See JimB's anwser for the complete example with WaitGroup.

Using the for range is also idiomatic if you want to use multiple worker goroutines: channels are synchronized so you don't need any extra code that would synchronize access to the todo channel or the jobs received from it. And if you close the todo channel in the main(), that will properly signal all worker goroutines. But of course all queued jobs will be received and processed exactly once.

Now taking the variant that uses WaitGroup to make the main goroutine to wait for the worker (JimB's answer): What if you want more than 1 worker goroutine; to process your jobs concurrently (and most likely parallel)?

The only thing you need to add / change in your code is this: to really start multiple of them:

for i := 0; i < 10; i++ {
    wg.Add(1)
    go ProcessToDo(todo)
}

Without changing anything else, you now have a correct, concurrent application which receives and processes your jobs using 10 concurrent goroutines. And we haven't used any "ugly" time.Sleep() (we used one but only to simulate slow processing, not to wait other goroutines), and you don't need any extra synchronization.

Answer 3

I think the accepted answer is pretty valid for this specific example. However to answer the question "Shutdown “worker” go routine after buffer is empty" - a more elegant solution is possible.

The worker can just return when the buffer is empty without needing to signal by closing the channel.

This is especially useful if the number of tasks that the worker needs to process in not known.

Check it out here: https://play.golang.org/p/LZ1y0eIRMeS

package main

import (
    "fmt"
    "time"
    "math/rand"
)

func main() {
    rand.Seed(time.Now().UnixNano())
    ch := make(chan interface{}, 10)

    go worker(ch)
    for i := 1; i <= rand.Intn(9) + 1; i++ {
            ch <- i
    }

    blocker := make(chan interface{})
    <-blocker
}

func worker(ch chan interface{}){   
    for {
        select {
        case msg := <- ch:
            fmt.Println("msg: ", msg)
        default:
            fmt.Println("exiting worker")
            return
        }
    }       
}