Raycast ignores all layers when trying to exclude just one

Question

I have the following raycast:

Ray ray1 = Camera.main.ScreenPointToRay(Input.mousePosition);
RaycastHit hit1;
if (Physics.Raycast(ray1, out hit1, ~layer))
{