Ok so I have a framebuffer with a bunch of attachments attached. The attachments are Color, Bloom, Velocity and Depth.
I start by clearing the framebuffer to the values
You cannot write to a color buffer conditionally from within a fragment shader.
An output fragment always has values for each active color buffer, even if you didn't write them. This is true even if you don't declare a variable for that output value. If no value gets written to a particular fragment output location, then the value used will be undefined. But it will be something.
You can do conditional writing based on color masking. But that is a per-draw-call thing, not something that can be done in the fragment shader. You could employ blending in that color buffer, using an alpha of 0 to mean "don't write", and an alpha of 1 to replace what's there.