How do I upgrade highly customised TestServer setup from .NET Core 3.0 to 3.1

Question

I have a TestServer that extends the AspNetCore.TestHost.TestServer that I am upgrading as a part of the .NET Core upgrade to 3.1, upgrading the projects works fine.