I was follwing the steps in the title "Add an allow list" in https://docs.microsoft.com/en-us/azure/active-directory/external-identities/allow-den