Convert number to human-readable KB/MB/GB/TB [duplicate]

Answer 1

Perhaps this is what you are looking for:

hdfs dfs -du -s /foo/bar/*tobedeleted | \
    sort -r -k 1 -g | \
    awk '{ suffix=" KMGT"; for(i=1; $1>1024 && i < length(suffix); i++) $1/=1024; print int($1) substr(suffix, i, 1), $3; }'

Answer 2

@innocent-bystander figured it out (just had to slightly modify his/her suggested solution):

$ hdfs dfs -du -s /foo/bar/* | sort -r -k 1 -g | head -5 | awk '{ suffix="KMGT"; for(i=0; $1>1024 && i < length(suffix); i++) $1/=1024; print int($1) substr(suffix, i, 1), $3; }' 
28T /foo/bar/card_dim_h_tobedeleted
20T /foo/bar/transaction_item_fct_tobedeleted
2T /foo/bar/card_dim_h_new_tobedeleted
2T /foo/bar/hshd_loyalty_seg_tobedeleted
1T /foo/bar/prod_dim_h_tobedeleted

(taking head also just to save some space on this page)

Thank you so much. Not only for solving this but also teaching me stuff I didn't know about awk. Very powerful isnt it?

Answer 3

You can use du with -h option to display data in human readable way hdfs dfs -du -s -h /user/vgunnu

Here is more info https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/FileSystemShell.html#du