How to get the realtime output for a shell command in golang?

Question

I am trying to call shell command with os/exec in golang, that command will take some time, so I would like to retrieve the reatime output and print the processed output (a

Answer 1

I do find icza's solution that he mentioned in that post is quite useful, however it didn't't solve my problem.

I did a little test as following:

1, I write a script which print some info every second for ten times, here is the script.sh

#!/bin/bash

for i in {1..10}
do
    echo "step " $i
    sleep 1s
done

2, read the stdout and extract the needed information from stdout and do some process to get the expected format, here is the code: package main

import (
    "fmt"
    "os/exec"
    "regexp"
    "strconv"
    "strings"
)

func getRatio(text string) float32 {
    re1, _ := regexp.Compile(`step[\s]+(\d+)`)
    result := re1.FindStringSubmatch(text)
    val, _ := strconv.Atoi(result[1])
    return float32(val) / 10
}

func main() {
    cmdName := "ffmpeg -i t.webm  -acodec aac -vcodec libx264  cmd1.mp4"
    //cmdName := "bash ./script.sh"
    cmdArgs := strings.Fields(cmdName)

    cmd := exec.Command(cmdArgs[0], cmdArgs[1:len(cmdArgs)]...)
    stdout, _ := cmd.StdoutPipe()
    cmd.Start()

    oneByte := make([]byte, 10)
    for {
        _, err := stdout.Read(oneByte)
        if err != nil {
            break
        }
        progressingRatio := getRatio(string(oneByte))
        fmt.Printf("progressing  ratio %v \n", progressingRatio)
    }
}

This does work for my script.sh test, but for the ffmpeg command it doesn't work, in ffmpeg's case, nothing get printed and the process get finished (not stuck), I guess the way of writing data to stdout for ffmpeg is a little special (maybe no newline character at all, and I tried icza's solution, but it still doesn't work).

Answer 2

check the below, needs enhancements (not recommended to be used as it is) but working :)

package main

import (
    "fmt"
    "os"
    "os/exec"
    "strconv"
    "strings"
)

var duration = 0
var allRes = ""
var lastPer = -1

func durToSec(dur string) (sec int) {
    durAry := strings.Split(dur, ":")
    if len(durAry) != 3 {
        return
    }
    hr, _ := strconv.Atoi(durAry[0])
    sec = hr * (60 * 60)
    min, _ := strconv.Atoi(durAry[1])
    sec += min * (60)
    second, _ := strconv.Atoi(durAry[2])
    sec += second
    return
}
func getRatio(res string) {
    i := strings.Index(res, "Duration")
    if i >= 0 {

        dur := res[i+10:]
        if len(dur) > 8 {
            dur = dur[0:8]

            duration = durToSec(dur)
            fmt.Println("duration:", duration)
            allRes = ""
        }
    }
    if duration == 0 {
        return
    }
    i = strings.Index(res, "time=")
    if i >= 0 {

        time := res[i+5:]
        if len(time) > 8 {
            time = time[0:8]
            sec := durToSec(time)
            per := (sec * 100) / duration
            if lastPer != per {
                lastPer = per
                fmt.Println("Percentage:", per)
            }

            allRes = ""
        }
    }
}

func main() {
    os.Remove("cmd1.mp4")
    cmdName := "ffmpeg -i 1.mp4  -acodec aac -vcodec libx264  cmd1.mp4 2>&1"
    cmd := exec.Command("sh", "-c", cmdName)
    stdout, _ := cmd.StdoutPipe()
    cmd.Start()
    oneByte := make([]byte, 8)
    for {
        _, err := stdout.Read(oneByte)
        if err != nil {
            fmt.Printf(err.Error())
            break
        }
        allRes += string(oneByte)
        getRatio(allRes)
    }
}

Answer 3

Looks like ffmpeg sends all diagnostic messages (the "console output") to stderr instead of stdout. Below code works for me.

package main

import (
    "bufio"
    "fmt"
    "os/exec"
    "strings"
)

func main() {
    args := "-i test.mp4 -acodec copy -vcodec copy -f flv rtmp://aaa/bbb"
    cmd := exec.Command("ffmpeg", strings.Split(args, " ")...)

    stderr, _ := cmd.StderrPipe()
    cmd.Start()

    scanner := bufio.NewScanner(stderr)
    scanner.Split(bufio.ScanWords)
    for scanner.Scan() {
        m := scanner.Text()
        fmt.Println(m)
    }
    cmd.Wait()
}

The version of ffmpeg is detailed as below.

ffmpeg version 3.0.2 Copyright (c) 2000-2016 the FFmpeg developers
built with Apple LLVM version 7.3.0 (clang-703.0.29)
configuration: --prefix=/usr/local/Cellar/ffmpeg/3.0.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-opencl --enable-libx264 --enable-libmp3lame --enable-libxvid --enable-vda
libavutil      55. 17.103 / 55. 17.103
libavcodec     57. 24.102 / 57. 24.102
libavformat    57. 25.100 / 57. 25.100
libavdevice    57.  0.101 / 57.  0.101
libavfilter     6. 31.100 /  6. 31.100
libavresample   3.  0.  0 /  3.  0.  0
libswscale      4.  0.100 /  4.  0.100
libswresample   2.  0.101 /  2.  0.101
libpostproc    54.  0.100 / 54.  0.100

Answer 4

When you have an exec.Cmd value of an external command you started from Go, you may use its Cmd.Stdin, Cmd.Stdout and Cmd.Stderr fields to communicate with the process in some way.

Some way means you can send data to its standard input, and you can read its standard output and error streams.

The stress is on standard. If the external process is sending data on a network connection, or is writing data to a file, you will not be able to intercept those data via the above mentioned 3 streams.

Now on to ffmpeg. ffmpeg and many other console applications do not write data to standard output/error, but they use system calls or other libraries (that use system calls) to manipulate the terminal window. Of course an application may send some data to the standard output/error, and may display other data by manipulating the terminal window.

So you don't see the output of ffmpeg because you try to read its standard output/error, but ffmpeg does not display its output by writing to those streams.

In the general case if you want to capture the output of such applications, you need a library that is capable of capturing the (textual) content of the terminal window. In an easier situation the application supports dumping those output to files usually controlled by extra command line parameters, which then you can read/monitor from Go.