GROUP BY/SUM from shell

Question

I have a large file containing data like this:

a 23
b 8
a 22
b 1

I want to be able to get this:

a 45
b 9

Answer 1

Edit: The modern (GNU/Linux) solution, as mentioned in comments years ago ;-) .

awk '{
    arr[$1]+=$2
   }
   END {
     for (key in arr) printf("%s\t%s\n", key, arr[key])
   }' file \
   | sort -k1,1

The originally posted solution, based on old Unix sort options:

awk '{
    arr[$1]+=$2
   }
   END {
     for (key in arr) printf("%s\t%s\n", key, arr[key])
   }' file \
   | sort +0n -1

I hope this helps.

Answer 2

This can be easily achieved with the following single-liner:

cat /path/to/file | termsql "SELECT col0, SUM(col1) FROM tbl GROUP BY col0"

Or.

termsql -i /path/to/file "SELECT col0, SUM(col1) FROM tbl GROUP BY col0"

Here a Python package, termsql, is used, which is a wrapper around SQLite. Note, that currently it's not upload to PyPI, and also can only be installed system-wide (setup.py is a little broken), like:

sudo pip install https://github.com/tobimensch/termsql/archive/master.zip

Answer 3

One way using perl:

perl -ane '
    next unless @F == 2; 
    $h{ $F[0] } += $F[1]; 
    END { 
        printf qq[%s %d\n], $_, $h{ $_ } for sort keys %h;
    }
' infile

Content of infile:

a 23
b 8
a 22
b 1

Output:

a 45
b 9

Answer 4

No need for awk here, or even sort -- if you have Bash 4.0, you can use associative arrays:

#!/bin/bash
declare -A values
while read key value; do
  values["$key"]=$(( $value + ${values[$key]:-0} ))
done
for key in "${!values[@]}"; do
  printf "%s %s\n" "$key" "${values[$key]}"
done

...or, if you sort the file first (which will be more memory-efficient; GNU sort is able to do tricks to sort files larger than memory, which a naive script -- whether in awk, python or shell -- typically won't), you can do this in a way which will work in older versions (I expect the following to work through bash 2.0):

#!/bin/bash
read cur_key cur_value
while read key value; do
  if [[ $key = "$cur_key" ]] ; then
    cur_value=$(( cur_value + value ))
  else
    printf "%s %s\n" "$cur_key" "$cur_value"
    cur_key="$key"
    cur_value="$value"
  fi
done
printf "%s %s\n" "$cur_key" "$cur_value"

Answer 5

This Perl one-liner seems to do the job:

perl -nle '($k, $v) = split; $s{$k} += $v; END {$, = " "; foreach $k (sort keys %s) {print $k, $s{$k}}}' inputfile

Answer 6

With GNU awk (versions less than 4):

WHINY_USERS= awk 'END {
  for (E in a)
    print E, a[E]
    }
{ a[$1] += $2 }' infile

With GNU awk >= 4:

awk 'END {
  PROCINFO["sorted_in"] = "@ind_str_asc"
  for (E in a)
    print E, a[E]
    }
{ a[$1] += $2 }' infile