Append to an array variable from a pipeline command

Question

I am writing a bash function to get all git repositories, but I have met a problem when I want to store all the git repository pathnames to the array patharray. Here is the code:

gitrepo() {
    local opt

    declare -a patharray
    locate -b '\.git' | \
        while read pathname
        do
            pathname="$(dirname ${pathname})"
            if [[ "${pathname}" != *.* ]]; then
            # Note: how to add an element to an existing Bash Array
                patharray=("${patharray[@]}" '\n' "${pathname}")
                # echo -e ${patharray[@]}
            fi
        done
    echo -e ${patharray[@]}
}

I want to save all the repository paths to the patharray array, but I can't get it outside the pipeline which is comprised of locate and while command.
But I can get the array in the pipeline command, the commented command # echo -e ${patharray[@]} works well if uncommented, so how can I solve the problem?

And I have tried the export command, however it seems that it can't pass the patharray to the pipeline.

Answer 1

First of all, appending to an array variable is better done with array[${#array[*]}]="value" or array+=("value1" "value2" "etc") unless you wish to transform the entire array (which you don't).

Now, since pipeline commands are run in subprocesses, changes made to a variable inside a pipeline command will not propagate to outside it. There are a few options to get around this (most are listed in Greg's BashFAQ/024):

• pass the result through stdout instead

  • the simplest; you'll need to do that anyway to get the value from the function (although there are ways to return a proper variable)

  • any special characters in paths can be handled reliably by using \0 as a separator (see Capturing output of find . -print0 into a bash array for reading \0-separated lists)

    locate -b0 '\.git' | while read -r -d '' pathname; do dirname -z "$pathname"; done
    

    or simply

    locate -b0 '\.git' | xargs -0 dirname -z
    

• avoid running the loop in a subprocess

  • avoid pipeline at all

    • temporary file/FIFO (bad: requires manual cleanup, accessible to others)
    • temporary variable (mediocre: unnecessary memory overhead)

    • process substitution (a special, syntax-supported case of FIFO, doesn't require manual cleanup; code adapted from Greg's BashFAQ/020):

      i=0    #`unset i` will error on `i' usage if the `nounset` option is set
      while IFS= read -r -d $'\0' file; do
        patharray[i++]="$(dirname "$file")"    # or however you want to process each file
      done < <(locate -b0 '\.git')
      

  • use the lastpipe option (new in Bash 4.2) - doesn't run the last command of a pipeline in a subprocess (mediocre: has global effect)

Answer 2

Bash runs all commands of a pipeline in separate SubShells. When a subshell containing a while loop ends, all changes you made to the patharray variable are lost.

You can simply group the while loop and the echo statement together so they are both contained within the same subshell:

gitrepo() {
    local pathname dir
    local -a patharray

    locate -b '\.git' | {                      # the grouping begins here
        while read pathname; do
            pathname=$(dirname "$pathname")
            if [[ "$pathname" != *.* ]]; then
                patharray+=( "$pathname" )     # add the element to the array
            fi
        done
        printf "%s\n" "${patharray[@]}"        # all those quotes are needed
    }                                          # the grouping ends here
}

Alternately, you can structure your code to not need a pipe: use ProcessSubstitution ( Also see the Bash manual for details - man bash | less +/Process\ Substitution):

gitrepo() {
    local pathname dir
    local -a patharray

    while read pathname; do
        pathname=$(dirname "$pathname")
        if [[ "$pathname" != *.* ]]; then
            patharray+=( "$pathname" )     # add the element to the array
        fi
    done < <(locate -b '\.git')

    printf "%s\n" "${patharray[@]}"        # all those quotes are needed
}