Keycloak behind nginx reverse proxy: SAML Integration invalid_destination

Question

I have keycloak on port 8081, proxied by nginx on 443 on a path /saml-sp. I followed all of the docs to set up the proxy. The well known open-id endpoint check and IP addres