I\'ve been doing a lot of studying and work recently related to WCF, web services and distributed computing in general, but most of the security concepts go over my head. Transp
Additionally, there's also the WCF Security Guidance by Microsoft's Patterns & Practices group. Check it out.
Marc
Start with searching wikipedia for Public Key Infrastructure (PKI) and follow the links to understand the different pieces. You don't need to know the encryption algorithims for the various ciphers, but you do need to understand the concepts if you want to really understand how WCF uses it.
Edit:
It's been several years since I first wrote the answer and the list is getting old. There have been some wide adoption of web-enabled APIs and token-based trust relaying.
I haven't read it, but Windows Communication Foundation Security would be a good place to start, if you're looking for something specific to WCF.
Also keep your eyes open for what major players like Facebook, Google, and Twitter are doing. They are using open protocols like OpenID and OAuth. At first, OAuth looks complicated, but you should understand the mechanism.
In my opinion earlier OAuth reinvents a lot of wheels that SSL has already solved, and leaves some security holes open. An interesting read is Compromising Twitter's OAuth security system. Facebook's OAuth 2.0 implementation and Google's OAuth 2.0 implementation simplify many of these issues by using https where it makes sense. These are must reads.
The basic concept around OAuth is trust relaying. You would want third-party developers to make apps against your API, but the end users cannot always trust these apps. Giving password to them, is like giving the keys to the kingdom. So the user types in the password into your UI, and your UI redirects to the third party with an access token.
Building Secure ASP.NET Applications: Authentication, Authorization, and Secure Communication is a good introduction to ASP.NET's security models. You can skip over the details because much of the technology is now obsolete.
A good overview specific to Web Services is Web Service Security: Scenarios, Patterns, and Implementation Guidance for Web Services Enhancements (WSE) 3.0. It says WSE, but basic concepts still remain the same.
To get more details on WS-Security, read Securing Web Services with WS-Security: Demystifying WS-Security, WS-Policy, SAML, XML Signature, and XML Encryption.
After reading above, what really helped me was looking at existing implementations like Amazon S3's authentication:
Amazon S3's authentication http://docs.amazonwebservices.com/AmazonS3/2006-03-01/images/HMACAuthProcess_You.gif
Flickr Authentication API:
Each authentication frob is specific to a user and an application's api key, and can only be used with that key.
Authentication frobs are valid for 60 minutes from the time it is created, or until the application calls flickr.auth.getToken, whichever is sooner.
Only one authentication frob per application per user will be valid at any one time. Applications must deal with expired and invalid authentication frobs and know how to renew them.
Twitter REST API
Many Twitter API methods require authentication. All responses are relative to the context of the authenticating user. For example, an attempt to retrieve information on a protected user who is not friends with the requesting user will fail.
For the time being, HTTP Basic Authentication is the only supported authentication scheme. When authenticating via Basic Auth, use your registered username or email address as the username component. Session cookies and parameter-based login are known to work but are not officially supported.
The OAuth token-based authentication scheme will shortly be offered as an experimental beta release.
So it's nice to know the complicated certs and PKI stuff, but the world seems to operate without it just fine.