Where is the lock for a std::atomic?

Question

If a data structure has multiple elements in it, the atomic version of it cannot (always) be lock-free. I was told that this is true for larger types because the CPU can not

Answer 1

The easiest way to answer such questions is generally to just look at the resulting assembly and take it from there.

Compiling the following (I made your struct larger to dodge crafty compiler shenanigans):

#include <atomic>

struct foo {
    double a;
    double b;
    double c;
    double d;
    double e;
};

std::atomic<foo> var;

void bar()
{
    var.store(foo{1.0,2.0,1.0,2.0,1.0});
}

In clang 5.0.0 yields the following under -O3: see on godbolt

bar(): # @bar()
  sub rsp, 40
  movaps xmm0, xmmword ptr [rip + .LCPI0_0] # xmm0 = [1.000000e+00,2.000000e+00]
  movaps xmmword ptr [rsp], xmm0
  movaps xmmword ptr [rsp + 16], xmm0
  movabs rax, 4607182418800017408
  mov qword ptr [rsp + 32], rax
  mov rdx, rsp
  mov edi, 40
  mov esi, var
  mov ecx, 5
  call __atomic_store

Great, the compiler delegates to an intrinsic (__atomic_store), that's not telling us what's really going on here. However, since the compiler is open source, we can easily find the implementation of the intrinsic (I found it in https://github.com/llvm-mirror/compiler-rt/blob/master/lib/builtins/atomic.c):

void __atomic_store_c(int size, void *dest, void *src, int model) {
#define LOCK_FREE_ACTION(type) \
    __c11_atomic_store((_Atomic(type)*)dest, *(type*)dest, model);\
    return;
  LOCK_FREE_CASES();
#undef LOCK_FREE_ACTION
  Lock *l = lock_for_pointer(dest);
  lock(l);
  memcpy(dest, src, size);
  unlock(l);
}

It seems like the magic happens in lock_for_pointer(), so let's have a look at it:

static __inline Lock *lock_for_pointer(void *ptr) {
  intptr_t hash = (intptr_t)ptr;
  // Disregard the lowest 4 bits.  We want all values that may be part of the
  // same memory operation to hash to the same value and therefore use the same
  // lock.  
  hash >>= 4;
  // Use the next bits as the basis for the hash
  intptr_t low = hash & SPINLOCK_MASK;
  // Now use the high(er) set of bits to perturb the hash, so that we don't
  // get collisions from atomic fields in a single object
  hash >>= 16;
  hash ^= low;
  // Return a pointer to the word to use
  return locks + (hash & SPINLOCK_MASK);
}

And here's our explanation: The address of the atomic is used to generate a hash-key to select a pre-alocated lock.

Answer 2

The usual implementation is a hash-table of mutexes (or even just simple spinlocks without a fallback to OS-assisted sleep/wakeup), using the address of the atomic object as a key. The hash function might be as simple as just using the low bits of the address as an index into a power-of-2 sized array, but @Frank's answer shows LLVM's std::atomic implementation does XOR in some higher bits so you don't automatically get aliasing when objects are separated by a large power of 2 (which is more common than any other random arrangement).

I think (but I'm not sure) that g++ and clang++ are ABI-compatible; i.e. that they use the same hash function and table, so they agree on which lock serializes access to which object. The locking is all done in libatomic, though, so if you dynamically link libatomic then all code inside the same program that calls __atomic_store_16 will use the same implementation; clang++ and g++ definitely agree on which function names to call, and that's enough. (But note that only lock-free atomic objects in shared memory between different processes will work: each process has its own hash table of locks. Lock-free objects are supposed to (and in fact do) Just Work in shared memory on normal CPU architectures, even if the region is mapped to different addresses.)

Hash collisions mean that two atomic objects might share the same lock. This is not a correctness problem, but it could be a performance problem: instead of two pairs of threads separately contending with each other for two different objects, you could have all 4 threads contending for access to either object. Presumably that's unusual, and usually you aim for your atomic objects to be lock-free on the platforms you care about. But most of the time you don't get really unlucky, and it's basically fine.

Deadlocks aren't possible because there aren't any std::atomic functions that try to take the lock on two objects at once. So the library code that takes the lock never tries to take another lock while holding one of these locks. Extra contention / serialization is not a correctness problem, just performance.

---

x86-64 16-byte objects with GCC vs. MSVC:

As a hack, compilers can use lock cmpxchg16b to implement 16-byte atomic load/store, as well as actual read-modify-write operations.

This is better than locking, but has bad performance compared to 8-byte atomic objects (e.g. pure loads contend with other loads). It's the only documented safe way to atomically do anything with 16 bytes¹.

AFAIK, MSVC never uses lock cmpxchg16b for 16-byte objects, and they're basically the same as a 24 or 32 byte object.

gcc6 and earlier inlined lock cmpxchg16b when you compile with -mcx16 (cmpxchg16b unfortunately isn't baseline for x86-64; first-gen AMD K8 CPUs are missing it.)

gcc7 decided to always call libatomic and never report 16-byte objects as lock-free, even though libatomic functions would still use lock cmpxchg16b on machines where the instruction is available. See is_lock_free() returned false after upgrading to MacPorts gcc 7.3. The gcc mailing list message explaining this change is here.

You can use a union hack to get a reasonably cheap ABA pointer+counter on x86-64 with gcc/clang: How can I implement ABA counter with c++11 CAS?. lock cmpxchg16b for updates of both pointer and counter, but simple mov loads of just the pointer. This only works if the 16-byte object is actually lock-free using lock cmpxchg16b, though.

---

Footnote 1: movdqa 16-byte load/store is atomic in practice on some (but not all) x86 microarchitectures, and there's no reliable or documented way to detect when it's usable. See Why is integer assignment on a naturally aligned variable atomic on x86?, and SSE instructions: which CPUs can do atomic 16B memory operations? for an example where K10 Opteron shows tearing at 8B boundaries only between sockets with HyperTransport.

So compiler writers have to err on the side of caution and can't use movdqa the way they use SSE2 movq for 8-byte atomic load/store in 32-bit code. It would be great if CPU vendors could document some guarantees for some microarchitectures, or add CPUID feature bits for atomic 16, 32, and 64-byte aligned vector load/store (with SSE, AVX, and AVX512). Maybe which mobo vendors could disable in firmware on funky many-socket machines that use special coherency glue chips that don't transfer whole cache lines atomically.

Answer 3

From 29.5.9 of the C++ standard:

Note: The representation of an atomic specialization need not have the same size as its corresponding argument type. Specializations should have the same size whenever possible, as this reduces the effort required to port existing code. — end note

It is preferable to make the size of an atomic the same as the size of its argument type, although not necessary. The way to achieve this is by either avoiding locks or by storing the locks in a separate structure. As the other answers have already explained clearly, a hash table is used to hold all the locks. This is the most memory efficient way of storing any number of locks for all the atomic objects in use.