I have been reading that Wp8.1 (XAML) apps are the new way of creating apps for Windows Phone 8.1, and the code is highly reusable for Windows 8.1 Desktop apps.
But
Are you comparing the same thing?
In the Silverlight version, you can only call sw.Stop in completion handler.
If you really want to do a good comparison, you should get an ETW trace; then you can really understand what's going up.
For Metro XAML based solution, there may be extra interop cost. But that seems to be the future path.
For Silverlight, existing API may be more polished for perf.
May be you should work on both solutions, make shareable code as big as possible, and later decide which way to take.