After having this issue in production for a long time, and having read anything i can find about (such as this or this or that), i made a simple test.
I have very similar problem, app is restarted after swapping to production slot and that causes unwanted downtime. After a lot of searching I found the following:
In some cases after the swap the web app in the production slot may restart later without any action taken by the app owner. This usually happens when the underlying storage infrastructure of Azure App Service undergoes some changes. When that happens the application will restart on all VMs at the same time which may result in a cold start and a high latency of the HTTP requests. While you cannot control the underlying storage events you can minimize the effect they have on your app in the production slot. Set this app setting on every slot of the app:
WEBSITE_ADD_SITENAME_BINDINGS_IN_APPHOST_CONFIG: setting this to “1” will prevent web app’s worker process and app domain from recycling when the App Service’s storage infrastructure gets reconfigured.
You can find the whole Ruslany post, that I found very helpful here