I am creating a publicly available API using API Gateway which is backed with lambda functions to do some processing. I have secured it with a custom security header that im
To prevent DDoS and higher rate of access, you can setup WAF. Have a look at this link, to get a deeper understanding how to setup WAF with API Gateway.
API Gateway will not charge you for unauthenticated requests, however you would be charged by Lambda for the invocation on the authorizer.
API Gateway offers a semi-useful mitigation to this problem in the form of the 'identity validation expression' on the Authorizer, which is just a regex that is matched against the incoming identity source header.
Besides that, you might want to just implement some kind of negative cache or validation yourself in the Authorizer function to minimize the billed milliseconds.