A simple chat mechanism will have 2 basic functionalities
Send the message to server (with info about the recipient)
Receive the message from server (designated for my user name)
First step is simple, we can create a web service which will accept the message with additional information about recipient(s). We can create it using any server side language.
Step 2, that is fetching the message from server can be done using 2 techniques, Pull the message (using polling) from server, or Push the message from server to android phone
Polling: In this, the android device will keep accessing server after a few seconds to check if there is a message available for user. This again can be implemented using a simple async task at the client side which will keep calling a web service after say 2-3 seconds. This is fine to use if we are planning to enable chatting only when user is accessing the app (no notifications like gmail or facebook), so that we can kill the polling service when not in use (otherwise it will eat up resources).
Push notifications: a better option is to use push notifications. Android provide Google cloud messaging or GCM (http://developer.android.com/google/gcm/index.html) which will help achieve push from server easily. Otherwise you can try a third party API like urbanairship or pushwoosh depending on your requirement. Push notifications will help the user to receive messages even when he is not using the app.
So in nutshell, a webservice to receive the messages and a push notification mechanism should be sufficient to implement a chat service in android.
Little bit about UrbanAirship
I used UA in one of my projects for push notifications as I needed to support both iOS and Android. If you just want to support Android GCM might also be a good option.
Coming back to UA, check this for sample code and usage: https://docs.urbanairship.com/display/DOCS/Home
The way it works is simple, when someone installs the app and is connected to internet, app registers itself to the UA service. A unique code is specified for each installed app (this is the time when you can capture the user name and unique code and store somewhere in your DB). Next UA provides an API using which you can push a message to designated recipient(s), using the unique codes which are available with UA. These messages can be received by android app and used as per the requirement. Even if the app is not running we can show a notification just like when we receive an email or a message
