Note: EboMike's accepted answer is INCORRECT.
I know this is an old question thread, but I stumbled across it trying to resolve the same problem. However the accepted answer was incorrect. There should be no need to run the adb server as root once you have your udev rules set correctly.
Step 3 in http://developer.android.com/tools/device.html gives you the correct resolution. Specifically, add or modify /etc/udev/rules.d/51-android.rules with the following line:
SUBSYSTEM=="usb", ATTR{idVendor}=="0bb4", ATTR{idProduct}=="0fff", MODE="0666", GROUP="plugdev"
In this case 0bb4 is the vendor id for HTC. 0fff is the product id for Nexus One. Use the table at the above link, or lsusb to identify your device's vendor id & product id. eg
$ lsusb
Bus 001 Device 006: ID 18d1:4e22 Google Inc. Nexus S (debug)
18d1 is the vendor id of this particular device, and 4e22 the product id. You can leave off the "GROUP" definition if you want to enable access to this device to all users. If not, be sure to add yourself to the plugdev group if you are not in it already. After creating this file, restart or reload udev.
udevadm control --reload-rules
If you had to add yourself to a group, you will need to logout and back in again.
Kill any old adb servers with "adb kill-server", plug your device in and run "adb devices". This will restart the server and you should now see your device. Running adb as root, even just to start the adb server, is not required. It is also generally a bad idea to run things as root unless it is absolutely required.