I\'m implementing a login/authentication system for my little server-client program. I\'m wondering how to go about this, and I was hoping to get some great tips/advice from Sta
You generally do not want to send the password over the link at all, not even with encryption. The usual method is a challenge-response protocol.
This has a couple of advantages. First, it means the password never goes over the link in any form. Second, it's immune to a replay attack -- if an attacker records the conversation, they can't replay the client's replies later to log in, because the random number will have changed.
Securing the connection (i.e., encrypting the content) is a little simpler. Typically, one of the two (doesn't really matter much which) picks a random number, encrypts it with the other's public key, and sends it to the other. The other decrypts it, and they encrypt the rest of the session using that as a key for symmetric encryption.
Libraries: Beecrypt and OpenSSL are a couple of obvious ones. Unless you have a fairly specific reason to do otherwise, TLS is what you probably want to use (it does quite a bit more than what I've outlined above, including two-way authentication, so not only does the server know who the client is, but the client also knows who the server is, so it's reasonably verified that it's not connected to somebody else who might just collect his credit card number and run with it).
Edit:
To authenticate each packet without the overhead of encrypting everything, you could do something like this:
Counter mode means you just generate consecutive numbers, and encrypt each in turn, using the right key. In this case, the key would be the hash of the client's password. What this means is that each packet will contain a unique random number that both the client and the server can generate, but nobody else can. By using the counter-mode encryption, each packet will have a unique random number. By starting from a random number, each session will have a unique sequence of random numbers.
To minimize overhead, you could send just a part of the result with each packet -- e.g., if you use AES in counter mode, it'll generate 16 bytes of result for each number you encrypt. Include only (say) two bytes of that with each packet, so you only have to encrypt a number once every 8 packets. In theory, this cuts security -- an attacker could just try all 65536 possible values for a packet, but if you assume the connection has been compromised after (say) two bad attempts, the chances of an attacker getting the right value become pretty small (and, of course, you can pretty much pick the chances you're willing to live with by controlling the number of bad attempts you allow and the size of authentication you include in each packet).