I would like to nest asp.net web app within sitecore site, something like this:
.
Right, your assumption is correct. See Creating a project in a virtual directory under Sitecore root article for more information about this kind of setup.
You should be able to create a virtual app under a Sitecore root path. Because its an app it will use its own app pool and you can have a web.config for it. I guess the physical files will be OUTSIDE of the Sitecore Website folder but the from a URL you can make the virtual app located anywhere below the Sitecore root.