While you can write your own HTTP server using normal TCP/IP (it's relatively simple), it is easier to use HttpListener, which takes advantage of the HTTP.SYS functionality added in Windows XP SP2.
However, HTTP.SYS adds the concept of URL ACLs. This is partly because HTTP.SYS allows you to bind to sub-namespaces on port 80. Using TCP/IP directly avoids this requirement, but means that you can't bind to a port that's already in use.
On Windows XP, you can use the HttpCfg.exe program to set up a URL ACL granting your user account the right to bind to a particular URL. It's in the Platform SDK samples.
On Windows Vista, HTTPCFG is still supported, but the functionality has been absorbed into NETSH:
netsh http show urlacl
...will show a list of existing URL ACLs. The ACLs are expressed in SDDL.
netsh http add urlacl url=http://+:80/MyUri user=DOMAIN\User listen=yes
...will configure the MyURI namespace so that DOMAIN\User can listen to requests.