For anyone else pulling their hair out, note the lines StartLimitInterval and StartLimitBurst in the caddy.service file - if you're testing and repeatedly stop / starting you will hit the StartLimitBurst limit. Need to increase it to lots, or change the StartLimitInterval, to allow this.