Captcha
They are annoying as hell, but most effective against detecting automated bots. If big sites like Google,Yahoo!,etc did not need captchas to detect bots then they would not use them because like you said they aren't very user-friendly.
OpenID
I keep repeating this but we don't need yet another username/password and I think you should be implementing openID instead to authenticate your users. I have made a library available at https://github.com/alfredwesterveld/php-openid which resembles a lot like the login which can be found at stackoverflow.com. If you want to first view a demo you could try openID on a little demo I made available at http://westerveld.name/php-openid/. When using OpenID the users probably has to perform captcha verification once to create an account.
Protecting against Bots
Spam
If for example you would like to detect if the comment is SPAM you could use something like akismet.
Akismet filters out your comment and
track-back spam for you, so you can
focus on more important things.
For personal blogs you can use this for free(or donate if you like product). For a small commercial blog you have to pay $5/month to detect spammy comments.
Other
Some other forms of verifying human would be:
- Sent email verification with unique link or something.
- Only allow users which you know are human. The rest should first proof they are users to you.
All these forms of verification could be broken and even CAPTCHA(especially simple ones) can be broken by really smart bots, but right now is the best solution against verifying.