I have a library that I use that uses WCF to call an http service to get settings. Normally the first call takes ~100 milliseconds and subsequent calls takes only a few millisec
Do you have an HTTP proxy defined in IE? (maybe an auto configure script). This can be a cause.
Otherwise I would guess it is the time that takes to load all the dlls. Try to deparate the proxy creation from the actull call to the service, to see what's taking the time.
I found the following article that talks about how only the first AppDomain can use native image dlls, so a child appdomain will always be forced to JIT lots of stuff that the initial AppDomain doesn't have to. This could lead to the performancce impact I am seeing, but would it be possible to somehow not get this performance penalty?
If there is a native image for the assembly, only the first AppDomain can use the native image. All other AppDomains will have to JIT-compile the code which can result in a significant CPU cost.
You can decorate your Main with LoaderOptimization attribute to tell the CLR loader how to load classes.
[LoaderOptimization(LoaderOptimization.MultiDomain)]
MultiDomain - Indicates that the application will probably have many domains that use the same code, and the loader must share maximal internal resources across application domains.