Quantifiable metrics (benchmarks) on the usage of header-only c++ libraries

Question

I\'ve tried to find an answer to this using SO. There are a number of questions that list the various pros and cons of building a header-only library in c++, but I haven\'t been

Answer 1

Update

This was Real Slaw's original answer. His answer above (the accepted one) is his second attempt. I feel that his second attempt answers the question entirely. - Homer6

Well, for comparison, you can look up the idea of "unity build" (nothing to do with the graphics engine). Basically, a "unity build" is where you include all the cpp files into a single file, and compile them all as one compilation unit. I think this should provide a good comparison, as AFAICT, this is equivalent to making your project header-only. You'd be surprised about the 2nd "con" you listed; the whole point of "unity builds" are to decrease compile times. Supposedly unity builds compile faster because they:

.. are a way of reducing build over-head (specifically opening and closing files and reducing link times by reducing the number of object files generated) and as such are used to drastically speed up build times.

― altdevblogaday

Compilation time comparison (from here):

Three major references for "unity build:

• http://buffered.io/posts/the-magic-of-unity-builds/
• http://cheind.wordpress.com/2009/12/10/reducing-compilation-time-unity-builds/
• http://www.altdevblogaday.com/2011/08/14/the-evils-of-unity-builds/

I assume you want reasons for the pros and cons listed.

Pros for header-only

[...]

3) It may be a lot faster. (quantifiable) The code might be optimized better. The reason is, when the units are separate, a function is just a function call, and thus must be left so. No information about this call is known, for example:

• Will this function modify memory (and thus our registers reflecting those variables/memory will be stale when it returns)?
• Does this function look at global memory (and thus we cannot reorder where we call the function)
• etc.

Furthermore, if the function internal code is known, it might be worthwhile to inline it (that is to dump its code directly into the calling function). Inlining avoids the function call overhead. Inlining also allows a whole host of other optimizations to occur (for example, constant propagation; for example we call factorial(10), now if the compiler doesn't know the code of factorial(), it is forced to leave it like that, but if we know the source code of factorial(), we can actually variables the variables in the function and replace it with 10, and if we are lucky we can even end up with the answer at compile time, without running anything at all at runtime). Other optimizations after inlining include dead-code elimination and (possibly) better branch prediction.

4) May give compiler/linker better opportunities for optimization (explanation/quantifiable, if possible)

I think this follows from (3).

Cons for header-only

1) It bloats the code. (quantifiable) (how does that affect both execution time and the memory footprint) Header-only can bloat the code in a few ways, that I know of.

The first is template bloat; where the compiler instantiates unnecessary templates of types that are never used. This isn't particular to header-only but rather templates, and modern compilers have improved on this to make it of minimal concern.

The second more obvious way, is the (over)inlining of functions. If a large function is inlined everywhere it is used, those calling functions will grow in size. This might have been a concern about executable size and executable-image-memory size years ago, but HDD space and memory have grown to make it almost pointless to care about. The more important issue is that this increased function size can ruin the instruction cache (so that the now-larger function doesn't fit into the cache, and now the cache has to be refilled as the CPU executes through the function). Register pressure will be increased after inlining (there is a limit on the number of registers, the on-CPU memory that the CPU can process with directly). This means that the compiler will have to juggle the registers in the middle of the now-larger-function, because there are too many variables.

2) Longer compile times. (quantifiable)

Well, header-only compilation can logically result in longer compile times for many reasons (notwithstanding the performance of "unity builds"; logic isn't necessarily real-world, where other factors get involved). One reason can be, if an entire project is header-only, then we lose incremental builds. This means any change in any part of the project means the entire project has to be rebuilt, while with separate compilation units, changes in one cpp just means that object file must be rebuilt, and the project relinked.

In my (anecdotal) experience, this is a big hit. Header-only increases performance a lot in some special cases, but productivity wise, it is usually not worth it. When you start getting a larger codebase, compilation time from scratch can take > 10 minutes each time. Recompiling on a tiny change starts getting tiresome. You don't know how many times I forgot a ";" and had to wait 5 mins to hear about it, only to go back and fix it, and then wait another 5 mins to find something else I just introduced by fixing the ";".

Performance is great, productivity is much better; it will waste a large chunk of your time, and demotivate/distract you from your programming goal.

Edit: I should mention, that interprocedural optimization (see also link-time optimization, and whole program optimization) tries to accomplish the optimization advantages of the "unity build". Implementations of this is still a bit shaky in most compilers AFAIK, but eventually this might overcome performance advantages.