Streaming commands output progress from Goroutine

Question

Streaming commands output progress question addresses the problem of printing progress of a long running command.

I tried to put the printing code within a goroutine

Answer 1

From godocs:

StdoutPipe returns a pipe that will be connected to the command's standard output when the command starts.

Wait will close the pipe after seeing the command exit, so most callers need not close the pipe themselves; however, an implication is that it is incorrect to call Wait before all reads from the pipe have completed.

You are calling Wait() immediately after starting the command. So the pipe gets closed as soon as the command completes, before making sure you have read all the data from the pipe. Try moving Wait() to your go routine after the scan loop.

go func(cmd *exec.Cmd, c chan int) {
    stdout, _ := cmd.StdoutPipe()

    <-c

    scanner := bufio.NewScanner(stdout)
    for scanner.Scan() {
        m := scanner.Text()
        fmt.Println(m)
    }

    cmd.Wait()
    c <- 1
}(cmd, c)

cmd.Start()
c <- 1

// This is here so we don't exit the program early,
<-c

There's also a simpler way to do things, which is to just assign os.stdout as the cmd's stdout, causing the command to directly write to the os.stdout:

cmd := exec.Command("some", "command")
cmd.Stdout = os.Stdout
cmd.Run()

Answer 2

There are some problems:

• The pipe is being closed before reading all data.
• Always check for errors
• Start cmd.Start() after c <- struct{}{} and use unbuffered channel c := make(chan struct{})

---

Two working sample codes:

1: Wait using channel then close the pipe after EOF using defer func() { c <- struct{}{} }(), like this working sample code:

package main

import (
    "bufio"
    "fmt"
    "os/exec"
)

func main() {
    cmd := exec.Command("Streamer")
    c := make(chan struct{})

    go run(cmd, c)

    c <- struct{}{}
    cmd.Start()

    <-c
    if err := cmd.Wait(); err != nil {
        fmt.Println(err)
    }
    fmt.Println("done.")
}

func run(cmd *exec.Cmd, c chan struct{}) {
    defer func() { c <- struct{}{} }()
    stdout, err := cmd.StdoutPipe()
    if err != nil {
        panic(err)
    }
    <-c
    scanner := bufio.NewScanner(stdout)
    for scanner.Scan() {
        m := scanner.Text()
        fmt.Println(m)
    }
    fmt.Println("EOF")
}

---

2: Also you may Wait using sync.WaitGroup, like this working sample code:

package main

import (
    "bufio"
    "fmt"
    "os/exec"
    "sync"
)

var wg sync.WaitGroup

func main() {
    cmd := exec.Command("Streamer")
    c := make(chan struct{})
    wg.Add(1)
    go func(cmd *exec.Cmd, c chan struct{}) {
        defer wg.Done()
        stdout, err := cmd.StdoutPipe()
        if err != nil {
            panic(err)
        }
        <-c
        scanner := bufio.NewScanner(stdout)
        for scanner.Scan() {
            m := scanner.Text()
            fmt.Println(m)
        }
    }(cmd, c)

    c <- struct{}{}
    cmd.Start()

    wg.Wait()
    fmt.Println("done.")
}

---

And Streamer sample code (just for testing):

package main

import "fmt"
import "time"

func main() {
    for i := 0; i < 10; i++ {
        time.Sleep(1 * time.Second)
        fmt.Println(i, ":", time.Now().UTC())
    }
}

---

And see func (c *Cmd) StdoutPipe() (io.ReadCloser, error) Docs:

StdoutPipe returns a pipe that will be connected to the command's standard output when the command starts.

Wait will close the pipe after seeing the command exit, so most callers need not close the pipe themselves; however, an implication is that it is incorrect to call Wait before all reads from the pipe have completed. For the same reason, it is incorrect to call Run when using StdoutPipe. See the example for idiomatic usage.