Ok guys just a small game:
I have some specifications for a project. At some point they ask for the following to encrypt a password over the net, saying that it is a cha
How about the following:
You are right -- if you capture the challenge and (challenge XOR password) then extracting the password is easy.
You need to use proper encryption in step 3, not XOR. Encrypt the challenge with the password.
To make an attacker's life harder you could add random data to what you encrypt to: e.g. encrypt paddingCHALLENGEpadding. The server doesn't care what the padding is, it knows where to look for the challenge, but it means an attacker won't know what the whole plaintext is.
You would be able to reverse engineer the password. You want to send the SHA of the password, not the password itself. Rolling your own security protocols is almost never a good idea. Can you not use SSL or something equivalent?
http://en.wikipedia.org/wiki/Cryptographic_nonce
That's a pretty horrible protocol. If this is something someone wants you to implement, refuse to. There are existing, vetted protocols for this type of thing. If this is a game where you point out all the flaws - okay.
And I'm definetly missing some more.