How can I get unique values from an array in Bash?

Question

I\'ve got almost the same question as here.

I have an array which contains aa ab aa ac aa ad, etc. Now I want to select all unique elements from this ar

Answer 1

If you're running Bash version 4 or above (which should be the case in any modern version of Linux), you can get unique array values in bash by creating a new associative array that contains each of the values of the original array. Something like this:

$ a=(aa ac aa ad "ac ad")
$ declare -A b
$ for i in "${a[@]}"; do b["$i"]=1; done
$ printf '%s\n' "${!b[@]}"
ac ad
ac
aa
ad

This works because in any array (associative or traditional, in any language), each key can only appear once. When the for loop arrives at the second value of aa in a[2], it overwrites b[aa] which was set originally for a[0].

Doing things in native bash can be faster than using pipes and external tools like sort and uniq, though for larger datasets you'll likely see better performance if you use a more powerful language like awk, python, etc.

If you're feeling confident, you can avoid the for loop by using printf's ability to recycle its format for multiple arguments, though this seems to require eval. (Stop reading now if you're fine with that.)

$ eval b=( $(printf ' ["%s"]=1' "${a[@]}") )
$ declare -p b
declare -A b=(["ac ad"]="1" [ac]="1" [aa]="1" [ad]="1" )

The reason this solution requires eval is that array values are determined before word splitting. That means that the output of the command substitution is considered a single word rather than a set of key=value pairs.

While this uses a subshell, it uses only bash builtins to process the array values. Be sure to evaluate your use of eval with a critical eye. If you're not 100% confident that chepner or glenn jackman or greycat would find no fault with your code, use the for loop instead.

Answer 2

# Read a file into variable
lines=$(cat /path/to/my/file)

# Go through each line the file put in the variable, and assign it a variable called $line
for line in $lines; do
  # Print the line
  echo $line
# End the loop, then sort it (add -u to have unique lines)
done | sort -u