Which is most cache friendly?

Question

I am trying to get a good grip on data oriented design and how to program best with the cache in mind. There\'s basically two scenarios that I cannot quite decide which is bette

Answer 1

Depends on your access patterns. Your first version is AoS (array of structures), second is SoA (structure of arrays).

SoA tends to use less memory (unless you store so few elements that the overhead of the arrays is actually non-trivial) if there's any kind of structure padding that you'd normally get in the AoS representation. It also tends to be a much bigger PITA to code against since you have to maintain/sync parallel arrays.

AoS tends to excel for random-access. As an example, for simplicity let's say each element fits into a cache line and is properly aligned (64 byte size and alignment, e.g.). In that case, if you are randomly accessing an nth element, you get all the relevant data for the element in a single cache line. If you used an SoA and dispersed those fields across separate arrays, you'd have to load memory into multiple cache lines just to load the data for that one element. And because we're accessing the data in a random pattern, we don't benefit from spatial locality much at all since the next element we're going to be accessing could be somewhere completely else in memory.

However, SoA tends to excel for sequential access mainly because there's often less data to load into the CPU cache in the first place for the entire sequential loop because it excludes structure padding and cold fields. By cold fields, I mean fields you don't need to access in a particular sequential loop. For example, a physics system might not care about particle fields involved with how the particle looks to the user, like color and a sprite handle. That's irrelevant data. It only cares about particle positions. The SoA allows you to avoid loading that irrelevant data into cache lines. It allows you to load as much relevant data into a cache line at once so you end up with fewer compulsory cache misses (as well as page faults for large enough data) with the SoA.

That's also only covering memory access patterns. With SoA reps, you also tend to be able to write more efficient and simpler SIMD instructions. But again it's mainly suited for sequential access.

You can also mix the two concepts. You might use an AoS for hot fields frequently accessed together in random-access patterns, then hoist out the cold fields and store them in parallel.