Memory efficient transpose - Awk

Question

i am trying to transpose a table (10k rows X 10K cols) using the following script.

A simple data example

$ cat rm1

        t1      t2      t         


        

Answer 1

There is a simple and quick algorithm based on sorting:

1) Make a pass through the input, prepending the row number and column number to each field. Output is a three-tuple of row, column, value for each cell in the matrix. Write the output to a temporary file.

2) Sort the temporary file by column, then row.

3) Make a pass through the sorted temporary file, reconstructing the transposed matrix.

The two outer passes are done by awk. The sort is done by the system sort. Here's the code:

$ echo '1 2 3
2 3 44
1 1 1' |
awk '{ for (i=1; i<=NF; i++) print i, NR, $i}' |
sort -n |
awk ' NR>1 && $2==1 { print "" }; { printf "%s ", $3 }; END { print "" }'
1 2 1
2 3 1
3 44 1