This is different for each browser, in IE and windows you need to create what they call a pluggable protocol handler.
The basic steps are as follows:
- Implement the IInternetProtocol interface.
- Implement the IInternetProtocolRoot interface.
- Implement the IClassFactory interface.
- Optional. Implement the IInternetProtocolInfo interface. Support for the HTTP protocol is provided by the transaction handler.
- If IInternetProtocolInfo is implemented, provide support for PARSE_SECURITY_URL and PARSE_SECURITY_DOMAIN so the URL security zone manager can handle the security properly.
Write the code for your protocol handler.
- Provide support for BINDF_NO_UI and BINDF_SILENTOPERATION.
- Add a subkey for your protocol handler in the registry under HKEY_CLASSES_ROOT\PROTOCOLS\Handler.
- Create a string value, CLSID, under the subkey and set the string to the CLSID of your protocol handler.
See About Asynchronous Pluggable Protocols on MSDN for more details on the windows side. There is also a sample in the windows SDK.
A quick google also showed this article on codeproject: http://www.codeproject.com/KB/IP/DataProtocol.aspx.
Finally, as a security guy I have to point out that this code needs to be battle hardened. It's at a high risk because to do it reliably you can't do it in managed code and have to do it in C++ (I suppose you could use VB6). You should consider whether you really need to do this and if you do, design it carefully and code it securely. An attacker can easily control the content that gets passed to you by simply including a link on a page. For example if you have a simple buffer overflow then nobody better do this: <a href="custom:foooo{insert long string for buffer overflow here}"> Click me for free porn</a>
Strongly consider using strsafe and the new secure CRT methods included in the VC8 and above compilers. See http://blogs.msdn.com/michael_howard/archive/2006/02/27/540123.aspx if you have no idea what I'm talking about.