GKE pods connection issues when having high load of requests

Question

I have a GKE v1.17.400 private cluster running with NAT gateway. On the cluster I have multiple application that use google services such as stackdriver, pubsub and cloud sql.