Azure has changed a fair bit from a year ago when I played with it.
I have a web app, and a Sql DB in my Azure resource group. I want my web app to read from my DB.
You don't need to store username / password details in the connection string with Azure Sql, you can add a token to the SqlConnection object. Here's a link to a site that details how to do it:
https://www.colinsalmcorner.com/post/configuring-aad-authentication-to-azure-sql-databases
Go to your Azure SQL Database and get a copy of the connection string. (It won't have your password, so you'll have to add that in a minute.
Next, go to your Web App, click on the "All Settings" and under then click "Application Settings".
Scroll down to "Connection Strings" and either create a new connection string (and paste it in from the first step) or update the existing connection string. Be sure to update the password in your connection string.
If you create a new connection string, make sure it's the same name as that one you are referencing in your code or you will have to change your code and redeploy as that as well.