My IIS7 web.config is set to the following with a folder of static assets (not within an ASP.NET app or anything):
You need to configure IIS to treat XAP as static content. Try this:
<configuration>
<system.webServer>
<staticContent>
<mimeMap fileExtension=".xaml" mimeType="application/xaml+xml" />
<mimeMap fileExtension=".xap" mimeType="application/x-silverlight-app" />
<mimeMap fileExtension=".xbap" mimeType="application/x-ms-xbap" />
</staticContent>
</system.webServer>
</configuration>