How does the compilation/linking process work?

Question

How does the compilation and linking process work?

_{(Note: This is meant to be an entry to Stack Overflow\'s C++ FAQ. If you want to critique the idea of}

Answer 1

The skinny is that a CPU loads data from memory addresses, stores data to memory addresses, and execute instructions sequentially out of memory addresses, with some conditional jumps in the sequence of instructions processed. Each of these three categories of instructions involves computing an address to a memory cell to be used in the machine instruction. Because machine instructions are of a variable length depending on the particular instruction involved, and because we string a variable length of them together as we build our machine code, there is a two step process involved in calculating and building any addresses.

First we laying out the allocation of memory as best we can before we can know what exactly goes in each cell. We figure out the bytes, or words, or whatever that form the instructions and literals and any data. We just start allocating memory and building the values that will create the program as we go, and note down anyplace we need to go back and fix an address. In that place we put a dummy to just pad the location so we can continue to calculate memory size. For example our first machine code might take one cell. The next machine code might take 3 cells, involving one machine code cell and two address cells. Now our address pointer is 4. We know what goes in the machine cell, which is the op code, but we have to wait to calculate what goes in the address cells till we know where that data will be located, i.e. what will be the machine address of that data.

If there were just one source file a compiler could theoretically produce fully executable machine code without a linker. In a two pass process it could calculate all of the actual addresses to all of the data cells referenced by any machine load or store instructions. And it could calculate all of the absolute addresses referenced by any absolute jump instructions. This is how simpler compilers, like the one in Forth work, with no linker.

A linker is something that allows blocks of code to be compiled separately. This can speed up the overall process of building code, and allows some flexibility with how the blocks are later used, in other words they can be relocated in memory, for example adding 1000 to every address to scoot the block up by 1000 address cells.

So what the compiler outputs is rough machine code that is not yet fully built, but is laid out so we know the size of everything, in other words so we can start to calculate where all of the absolute addresses will be located. the compiler also outputs a list of symbols which are name/address pairs. The symbols relate a memory offset in the machine code in the module with a name. The offset being the absolute distance to the memory location of the symbol in the module.

That's where we get to the linker. The linker first slaps all of these blocks of machine code together end to end and notes down where each one starts. Then it calculates the addresses to be fixed by adding together the relative offset within a module and the absolute position of the module in the bigger layout.

Obviously I've oversimplified this so you can try to grasp it, and I have deliberately not used the jargon of object files, symbol tables, etc. which to me is part of the confusion.