Assembling 32-bit binaries on a 64-bit system (GNU toolchain)

Question

I wrote assembly code that successfully compiles:

as power.s -o power.o

However, it fails when I try to link the object file:

Answer 1

TL:DR: use gcc -m32 -static -nostdlib foo.S (or equivalent as and ld options).
Or if you don't define your own _start, just gcc -m32 -no-pie foo.S

You may need to install gcc-multilib if you link libc, or however your distro packages /usr/lib32/libc.so, /usr/lib32/libstdc++.so and so on. But if you define your own _start and don't link libraries, you don't need the library package, just a kernel that supports 32-bit processes and system calls. This includes most distros, but not Windows Subsystem for Linux v1.

Don't use .code32

.code32 does not change the output file format, and that's what determines the mode your program will run in. It's up to you to not try to run 32bit code in 64bit mode. .code32 is for assembling kernels that have some 16 and some 32-bit code, and stuff like that. If that's not what you're doing, avoid it so you'll get build-time errors when you build a .S in the wrong mode if it has any push or pop instructions, for example. .code32 just lets you create confusing-to-debug runtime problems instead of build-time errors.

Suggestion: use the .S extension for hand-written assembler. (gcc -c foo.S will run it through the C preprocessor before as, so you can #include for syscall numbers, for example). Also, it distinguishes it from .s compiler output (from gcc foo.c -O3 -S).

To build 32-bit binaries, use one of these commands

gcc -g foo.S -o foo -m32 -nostdlib -static  # static binary with absolutely no libraries or startup code
                       # -nostdlib still dynamically links when Linux where PIE is the default, or on OS X

gcc -g foo.S -o foo -m32 -no-pie            # dynamic binary including the startup boilerplate code.
     # Use with code that defines a main(), not a _start

Documentation for nostdlib, -nostartfiles, and -static.

---

Using libc functions from _start (see the end of this answer for an example)

Some functions, like malloc(3), or stdio functions including printf(3), depend on some global data being initialized (e.g. FILE *stdout and the object it actually points to).

gcc -nostartfiles leaves out the CRT _start boilerplate code, but still links libc (dynamically, by default). On Linux, shared libraries can have initializer sections that are run by the dynamic linker when it loads them, before jumping to your _start entry point. So gcc -nostartfiles hello.S still lets you call printf. For a dynamic executable, the kernel runs /lib/ld-linux.so.2 on it instead of running it directly (use readelf -a to see the "ELF interpreter" string in your binary). When your _start eventually runs, not all the registers will be zeroed, because the dynamic linker ran code in your process.

However, gcc -nostartfiles -static hello.S will link, but crash at runtime if you call printf or something without calling glibc's internal init functions. (see Michael Petch's comment).

---

Of course you can put any combination of .c, .S, and .o files on the same command line to link them all into one executable. If you have any C, don't forget -Og -Wall -Wextra: you don't want to be debugging your asm when the problem was something simple in the C that calls it that the compiler could have warned you about.

Use -v to have gcc show you the commands it runs to assemble and link. To do it "manually":

as foo.S -o foo.o -g --32 &&      # skips the preprocessor
ld -o foo foo.o  -m elf_i386

file foo
foo: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, not stripped

gcc -nostdlib -m32 is easier to remember and type than the two different options for as and ld (--32 and -m elf_i386). Also, it works on all platforms, including ones where executable format isn't ELF. (But Linux examples won't work on OS X, because the system call numbers are different, or on Windows because it doesn't even use the int 0x80 ABI.)

---

NASM/YASM

gcc can't handle NASM syntax. (-masm=intel is more like MASM than NASM syntax, where you need offset symbol to get the address as an immediate). And of course the directives are different (e.g. .globl vs global).

You can build with nasm or yasm, then link the .o with gcc as above, or ld directly.

I use a wrapper script to avoid the repetitive typing of the same filename with three different extensions. (nasm and yasm default to file.asm -> file.o, unlike GNU as's default output of a.out). Use this with -m32 to assemble and link 32bit ELF executables. Not all OSes use ELF, so this script is less portable than using gcc -nostdlib -m32 to link would be..

#!/bin/bash
# usage: asm-link [-q] [-m32] foo.asm  [assembler options ...]
# Just use a Makefile for anything non-trivial.  This script is intentionally minimal and doesn't handle multiple source files
# Copyright 2020 Peter Cordes.  Public domain.  If it breaks, you get to keep both pieces

verbose=1                       # defaults
fmt=-felf64
#ldopt=-melf_i386
ldlib=()

linker=ld
#dld=/lib64/ld-linux-x86-64.so.2
while getopts 'Gdsphl:m:nvqzN' opt; do
    case "$opt" in
        m)  if [ "m$OPTARG" = "m32" ]; then
                fmt=-felf32
                ldopt=-melf_i386
                #dld=/lib/ld-linux.so.2  # FIXME: handle linker=gcc non-static executable
            fi
            if [ "m$OPTARG" = "mx32" ]; then
                fmt=-felfx32
                ldopt=-melf32_x86_64
            fi
            ;;
        #   -static
        l)  linker="gcc -no-pie -fno-plt -nostartfiles"; ldlib+=("-l$OPTARG");;
        p)  linker="gcc -pie -fno-plt -nostartfiles"; ldlib+=("-pie");;
        h)  ldlib+=("-Ttext=0x200800000");;   # symbol addresses outside the low 32.  data and bss go in range of text
                          # strace -e raw=write  will show the numeric address
        G)  nodebug=1;;      # .label: doesn't break up objdump output
        d)  disas=1;;
        s)  runsize=1;;
        n)  use_nasm=1 ;;
        q)  verbose=0 ;;
        v)  verbose=1 ;;
        z)  ldlib+=("-zexecstack") ;;
        N)  ldlib+=("-N") ;;   # --omagic = read+write text section
    esac
done
shift "$((OPTIND-1))"   # Shift off the options and optional --

src=$1
base=${src%.*}
shift

#if [[ ${#ldlib[@]} -gt 0 ]]; then
    #    ldlib+=("--dynamic-linker" "$dld")
    #ldlib=("-static" "${ldlib[@]}")
#fi

set -e
if (($use_nasm)); then
  #  (($nodebug)) || dbg="-g -Fdwarf"     # breaks objdump disassembly, and .labels are included anyway
    ( (($verbose)) && set -x    # print commands as they're run, like make
    nasm "$fmt" -Worphan-labels $dbg  "$src" "$@" &&
        $linker $ldopt -o "$base" "$base.o"  "${ldlib[@]}")
else
    (($nodebug)) || dbg="-gdwarf2"
    ( (($verbose)) && set -x    # print commands as they're run, like make
    yasm "$fmt" -Worphan-labels $dbg "$src" "$@" &&
        $linker $ldopt -o "$base" "$base.o"  "${ldlib[@]}" )
fi

# yasm -gdwarf2 includes even .local labels so they show up in objdump output
# nasm defaults to that behaviour of including even .local labels

# nasm defaults to STABS debugging format, but -g is not the default

if (($disas));then
    objdump -drwC -Mintel "$base"
fi

if (($runsize));then
    size $base
fi

I prefer YASM for a few reasons, including that it defaults to making long-nops instead of padding with many single-byte nops. That makes for messy disassembly output, as well as being slower if the nops ever run. (In NASM, you have to use the smartalign macro package.)

However, YASM hasn't been maintained for a while and only NASM has AVX512 support; these days I more often just use NASM.

---

Example: a program using libc functions from _start

# hello32.S

#include    // syscall numbers.  only #defines, no C declarations left after CPP to cause asm syntax errors

.text
#.global main   # uncomment these to let this code work as _start, or as main called by glibc _start
#main:
#.weak _start

.global _start
_start:
        mov     $__NR_gettimeofday, %eax  # make a syscall that we can see in strace output so we know when we get here
        int     $0x80

        push    %esp
        push    $print_fmt
        call   printf

        #xor    %ebx,%ebx                 # _exit(0)
        #mov    $__NR_exit_group, %eax    # same as glibc's _exit(2) wrapper
        #int    $0x80                     # won't flush the stdio buffer

        movl    $0, (%esp)   # reuse the stack slots we set up for printf, instead of popping
        call    exit         # exit(3) does an fflush and other cleanup

        #add    $8, %esp     # pop the space reserved by the two pushes
        #ret                 # only works in main, not _start

.section .rodata
print_fmt: .asciz "Hello, World!\n%%esp at startup = %#lx\n"

---

$ gcc -m32 -nostdlib hello32.S
/tmp/ccHNGx24.o: In function `_start':
(.text+0x7): undefined reference to `printf'
...
$ gcc -m32 hello32.S
/tmp/ccQ4SOR8.o: In function `_start':
(.text+0x0): multiple definition of `_start'
...

---

Fails at run-time, because nothing calls the glibc init functions. (__libc_init_first, __dl_tls_setup, and __libc_csu_init in that order, according to Michael Petch's comment. Other libc implementations exist, including MUSL which is designed for static linking and works without initialization calls.)

$ gcc -m32 -nostartfiles -static hello32.S     # fails at run-time
$ file a.out
a.out: ELF 32-bit LSB executable, Intel 80386, version 1 (GNU/Linux), statically linked, BuildID[sha1]=ef4b74b1c29618d89ad60dbc6f9517d7cdec3236, not stripped
$ strace -s128 ./a.out
execve("./a.out", ["./a.out"], [/* 70 vars */]) = 0
[ Process PID=29681 runs in 32 bit mode. ]
gettimeofday(NULL, NULL)                = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=0} ---
+++ killed by SIGSEGV (core dumped) +++
Segmentation fault (core dumped)

You could also gdb ./a.out, and run b _start, layout reg, run, and see what happens.

---

$ gcc -m32 -nostartfiles hello32.S             # Correct command line
$ file a.out
a.out: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, BuildID[sha1]=7b0a731f9b24a77bee41c13ec562ba2a459d91c7, not stripped

$ ./a.out
Hello, World!
%esp at startup = 0xffdf7460

$ ltrace -s128 ./a.out > /dev/null
printf("Hello, World!\n%%esp at startup = %#lx\n", 0xff937510)      = 43    # note the different address: Address-space layout randomization at work
exit(0 
+++ exited (status 0) +++

$ strace -s128 ./a.out > /dev/null        # redirect stdout so we don't see a mix of normal output and trace output
execve("./a.out", ["./a.out"], [/* 70 vars */]) = 0
[ Process PID=29729 runs in 32 bit mode. ]
brk(0)                                  = 0x834e000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
....   more syscalls from dynamic linker code
open("/lib/i386-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
mmap2(NULL, 1814236, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xfffffffff7556000    # map the executable text section of the library
... more stuff
# end of dynamic linker's code, finally jumps to our _start

gettimeofday({1461874556, 431117}, NULL) = 0
fstat64(1, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0  # stdio is figuring out whether stdout is a terminal or not
ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0xff938870) = -1 ENOTTY (Inappropriate ioctl for device)
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xfffffffff7743000      # 4k buffer for stdout
write(1, "Hello, World!\n%esp at startup = 0xff938fb0\n", 43) = 43
exit_group(0)                           = ?
+++ exited with 0 +++

If we'd used _exit(0), or made the sys_exit system call ourselves with int 0x80, the write(2) wouldn't have happened. With stdout redirected to a non-tty, it defaults to full-buffered (not line-buffered), so the write(2) is only triggered by the fflush(3) as part of exit(3). Without redirection, calling printf(3) with a string containing newlines will flush immediately.

Behaving differently depending on whether stdout is a terminal can be desirable, but only if you do it on purpose, not by mistake.