Converting hex to decimal in awk or sed

Question

I have a list of numbers, comma-separated:

123711184642,02,3583090366663629,639f02012437d4
123715942138,01,3538710295145500,639f02afd6c643
123711616258,02,35         


        

Answer 1

Perl version, with a tip of the hat to @Jonathan:

perl -F, -lane '$p1 = substr($F[3], 0, 6); $p2 = substr($F[3], 6, 4); $p3 = substr($F[3], 10, 4); printf "%s,%s,%s,%s,%d,%d\n", @F[0..2], $p1, hex($p2), hex($p3)' file

-a turn on autosplit mode, to populate the @F array
-F, changes the autosplit separator to , (default is whitespace)
The substr() indices are 1 less than their awk equivalents, since Perl arrays start from 0.

Output:

123711184642,02,3583090366663629,639f02,292,14292
123715942138,01,3538710295145500,639f02,45014,50755
123711616258,02,3548370476972758,639f02,72,22322

Answer 2

By AWK

This answer concentrates on showing how to do the conversion by awk portably.

Using --non-decimal-data for gawk is not recommended according to GNU Awk User's Guide. And using strtonum() is not portable.

In the following examples the first word of each record is converted.

By user-defined function

The most portable way of doing conversion is by a user-defined awk function [reference]:

function parsehex(V,OUT)
{
    if(V ~ /^0x/)  V=substr(V,3);

    for(N=1; N<=length(V); N++)
        OUT=(OUT*16) + H[substr(V, N, 1)]

    return(OUT)
}

BEGIN { for(N=0; N<16; N++)
        {  H[sprintf("%x",N)]=N; H[sprintf("%X",N)]=N } }

{ print parsehex($1) }

By calling shell's printf

You could use this

awk '{cmd="printf %d 0x" $1; cmd | getline decimal; close(cmd); print decimal}'

but it is relatively slow. The following one is faster, if you have many newline-separated hexadecimal numbers to convert:

awk 'BEGIN{cmd="printf \"%d\n\""}{cmd=cmd " 0x" $1}END{while ((cmd | getline dec) > 0) { print dec }; close(cmd)}'

There might be a problem if very many arguments are added for the single printf command.

In Linux

In my experience the following works in Linux:

awk -Wposix '{printf("%d\n","0x" $1)}'

I tested it by gawk, mawk and original-awk in Ubuntu Linux 14.04. By original-awk the command displays a warning message, but you can hide it by redirection directive 2>/dev/null in shell. If you don't want to do that, you can strip the -Wposix in case of original-awk like this:

awk $(awk -Wversion >/dev/null 2>&1 && printf -- "-Wposix") '{printf("%d\n","0x" $1)}'

(In Bash 4 you could replace >/dev/null 2>&1 by &>/dev/null)

Note: The -Wposix trick probably doesn't work with nawk which is used in OS X and some BSD OS variants, though.

Answer 3

This might work for you (GNU sed & printf):

sed -r 's/(....)(....)$/ 0x\1 0x\2/;s/.*/printf "%s,%d,%d" &/e' file

Split the last eight characters and add spaces preceeding the fields by the hex identifier and then evaluate the whole line using printf.

Answer 4

cat all_info_List.csv| awk 'BEGIN {FS="|"}{print $21}'| awk 'BEGIN {FS=":"}{p1=$1":"$2":"$3":"$4":"$5":";  p2 = strtonum("0x"$6); printf("%s%02X\n",p1,p2+1) }'

The above command prints the contents of "all_info_List.csv", a file where the field separator is "|". Then takes field 21 (MAC address) and splits it using field separator ":". It assigns to variable "p1" the first 5 bytes of each mac address, so if we had this mac address:"11:22:33:44:55:66", p1 would be: "11:22:33:44:55:". p2 is assigned with the decimal value of the last byte: "0x66" would assign "102" decimal to p2. Finally, I'm using printf to join p1 and p2, while converting p2 back to hex, after adding one to it.

Answer 5

This seems to work:

awk -F, '{ p1 =       substr($4,  1, 6);
           p2 = ("0x" substr($4,  7, 4)) + 0;
           p3 = ("0x" substr($4, 11, 4)) + 0;
           printf "%s,%s,%s,%s,%d,%d\n", $1, $2, $3, p1, p2, p3;
         }'

For your sample input data, it produces:

123711184642,02,3583090366663629,639f02,292,14292
123715942138,01,3538710295145500,639f02,45014,50755
123711616258,02,3548370476972758,639f02,72,22322

The string concatenation of '0x' plus the 4-digit hex followed by adding 0 forces awk to treat the numbers as hexadecimals.

You can simplify this to:

awk -F, '{ p1 =      substr($4,  1, 6);
           p2 = "0x" substr($4,  7, 4);
           p3 = "0x" substr($4, 11, 4);
           printf "%s,%s,%s,%s,%d,%d\n", $1, $2, $3, p1, p2, p3;
         }'

The strings prefixed with 0x are forced to integer when presented to printf() and the %d format.

---

The code above works beautifully with the native awk on MacOS X 10.6.5 (version 20070501); sadly, it does not work with GNU gawk 3.1.7. That, it seems, is permitted behaviour according to POSIX (see the comments below). However, gawk has a non-standard function strtonum that can be used to bludgeon it into performing correctly - pity that bludgeoning is necessary.

gawk -F, '{ p1 =      substr($4,  1, 6);
            p2 = "0x" substr($4,  7, 4);
            p3 = "0x" substr($4, 11, 4);
            printf "%s,%s,%s,%s,%d,%d\n", $1, $2, $3, p1, strtonum(p2), strtonum(p3);
          }'

Answer 6

Here's a variation on Jonathan's answer:

awk $([[ $(awk --version) = GNU* ]] && echo --non-decimal-data) -F, '
    BEGIN {OFS = FS}
    {
        $6 = sprintf("%d", "0x" substr($4, 11, 4))
        $5 = sprintf("%d", "0x" substr($4,  7, 4))
        $4 = substr($4,  1, 6)
        print
    }'

I included a rather contorted way of adding the --non-decimal-data option if it's needed.

Edit

Just for the heck of it, here's the pure-Bash equivalent:

saveIFS=$IFS
IFS=,
while read -r -a line
do
    printf '%s,%s,%d,%d\n' "${line[*]:0:3}" "${line[3]:0:6}" "0x${line[3]:6:4}" "0x${line[3]:10:4}"
done
IFS=$saveIFS

The "${line[*]:0:3}" (quoted *) works similarly to AWK's OFS in that it causes Bash's IFS (here a comma) to be inserted between array elements on output. We can take further advantage of that feature by inserting array elements as follows which more closely parallels my AWK version above.

saveIFS=$IFS
IFS=,
while read -r -a line
do
    line[6]=$(printf '%d' "0x${line[3]:10:4}")
    line[5]=$(printf '%d' "0x${line[3]:6:4}")
    line[4]=$(printf '%s' "${line[3]:0:6}")
    printf '%s\n' "${line[*]}"
done
IFS=$saveIFS

Unfortunately, Bash doesn't allow printf -v (which is similar to sprintf()) to make assignments to array elements, so printf -v "line[6]" ... doesn't work.

Edit: As of Bash 4.1, printf -v can now make assignments to array elements. Example:

printf -v 'line[6]' '%d' "0x${line[3]:10:4}"

The quotes around the array reference are needed to prevent possible filename matching. If a file named "line6" existed in the current directory and the reference wasn't quoted, then a variable named line6 would be created (or updated) containing the printf output. Nothing else about the file, such as its contents, would come into play. Only the name - and only tangentially.