How can I get unique values from an array in Bash?

Question

I\'ve got almost the same question as here.

I have an array which contains aa ab aa ac aa ad, etc. Now I want to select all unique elements from this ar

Answer 1

Try this to get uniq values for first column in file

awk -F, '{a[$1];}END{for (i in a)print i;}'

Answer 2

To create a new array consisting of unique values, ensure your array is not empty then do one of the following:

Remove duplicate entries (with sorting)

readarray -t NewArray < <(printf '%s\n' "${OriginalArray[@]}" | sort -u)

Remove duplicate entries (without sorting)

readarray -t NewArray < <(printf '%s\n' "${OriginalArray[@]}" | awk '!x[$0]++')

Warning: Do not try to do something like NewArray=( $(printf '%s\n' "${OriginalArray[@]}" | sort -u) ). It will break on spaces.

Answer 3

How about this variation?

printf '%s\n' "${ids[@]}" | sort -u

Answer 4

I realize this was already answered, but it showed up pretty high in search results, and it might help someone.

printf "%s\n" "${IDS[@]}" | sort -u

Example:

~> IDS=( "aa" "ab" "aa" "ac" "aa" "ad" )
~> echo  "${IDS[@]}"
aa ab aa ac aa ad
~>
~> printf "%s\n" "${IDS[@]}" | sort -u
aa
ab
ac
ad
~> UNIQ_IDS=($(printf "%s\n" "${IDS[@]}" | sort -u))
~> echo "${UNIQ_IDS[@]}"
aa ab ac ad
~>

Answer 5

'sort' can be used to order the output of a for-loop:

for i in ${ids[@]}; do echo $i; done | sort

and eliminate duplicates with "-u":

for i in ${ids[@]}; do echo $i; done | sort -u

Finally you can just overwrite your array with the unique elements:

ids=( `for i in ${ids[@]}; do echo $i; done | sort -u` )

Answer 6

cat number.txt

1 2 3 4 4 3 2 5 6

print line into column: cat number.txt | awk '{for(i=1;i<=NF;i++) print $i}'

1
2
3
4
4
3
2
5
6

find the duplicate records: cat number.txt | awk '{for(i=1;i<=NF;i++) print $i}' |awk 'x[$0]++'

4
3
2

Replace duplicate records: cat number.txt | awk '{for(i=1;i<=NF;i++) print $i}' |awk '!x[$0]++'

1
2
3
4
5
6

Find only Uniq records: cat number.txt | awk '{for(i=1;i<=NF;i++) print $i|"sort|uniq -u"}

1
5
6