On my local machine, I work on multiple web sites and run them under IIS under a \"Default\" web site. That way I can access the sites through this type of URL: http://localhos
I see this problem once in a while and it's always been that the server isn't recognizing it as a configured web. What you could try if possible is remove the application and then convert each sub-directory back to it's own application.
This has always fixed it for me. Double and triple check that the sub directories are set as applications.
Also to avoid other problems, since you're setting Windows authentication check the NTLM vs Kerberos settings. Usually this is a different error message or just won't authenticate properly, but if this is an internet environment instead of an intranet environment you'll most likely need to disable Kerberos.
Also make sure you've checked that Windows Authentication is enabled. http://technet.microsoft.com/en-us/library/cc754628(WS.10).aspx
You can enable and disable Kerberos under the Advanced Settings within Windows Authentication.