I\'m trying to wrap my mind around Amazon\'s Server Side Encryption options so I can start asking S3 to encrypt my data at rest when my applications upload files.
So far
The answer from Michael is a good explanation of envelope encryption and rotation of unbacked master key material.
Unfortunately, that's not how AWS works. When you rotate master keys, AWS NEVER re-encrypts any data keys (or data). If you are rotating a CMK generated using AWS key material, a new backing key is simply added to the existing CMK. The AWS-managed CMK is made up of the current backing key and ALL historical backing keys (which are used to decrypt old, rotated data keys). AWS never throws away HBK backing material unless the entire keychain is deleted by the customer. Every time you rotate the keys, the CMK accumulates more HBK material.
Imported master keys do not support key backing. If you are manually rotating a CMK that uses customer imported key material, you must keep your old keys if you wish to decrypt old data that was encrypted with this key prior to rotation. You can associate the new key with the old key alias to switch security contexts within your application without making any code changes.
Following your analogy above, AWS never replaces the lockbox. It creates another (new) lockbox right next to the old one (and pretends they are both 'the lockbox'). It continues to use the keys in the old lockbox to open existing apartments, while new keys get put in the new lockbox. When you instruct it to 'delete', it destroys all the associated lockboxes.
You can read the full details on pages 11 and 12 of the AWS "Cryptographic Details" whitepaper, here:
https://d0.awsstatic.com/whitepapers/KMS-Cryptographic-Details.pdf