Here is my scenario.
I have my device (android phone). And I know the IP address of another device (which is some remote device in the Internet)
So how do I sen
In most realistic cases, you send data to the other device by sending the data to an intermediate device from which the other phone picks it up. Can can use protocols like TCP or UDP directly between the two devices, and that may or may not work depending on the protocols and how the devices are getting Internet access.
For example, one device can send an email and the other device can retrieve that email. You can generalize that model to other types of communication.
What you're looking to do is a common task with a common problem (but also with a common solution). You want to send data from one device (computer/phone/etc..) to another. Here's how it works behind the scenes:
There are a lot of computers in the world. Each computer is given an identity - a name, if you will, which is called the computer's IP address. There are many versions/standards of the IP address, and the current standard names an IP address as such: "111.222.333.444". There are only so many permutations with 12 digits of numbers - and entire ranges of IP addresses are reserved. In other words, there aren't enough IP addresses to go around for computers.
Now, routers were invented, among other reasons, to combat this problem. Routers give all computers within its local area network (LAN) one public (WAN) IP address. With an average of one router per household, this saves many IP addresses. Unfortunately, it also breaks direct communication between computers, because how does one communicate with three computers all using the same name? How does one identify computer A from computer B from computer C when they all share the same WAN IP address?
First, let's clarify that - just like there are a few versions and standards of IP addresses, so there are a few types of routers. By the way, it's not the router that is to blame for the problems of direct communication between computers - it's actually the router's underlying technology of network address translation (NAT). Anyways, there are a few types of NAT, and although information on the internet is relatively scarce on these types, you can still Wikipedia it and read a few university research papers. The types are Full Cone NAT, Restricted NAT, Port Restricted NAT, and Symmetric NAT.
So, computers in a LAN share the same WAN IP. This is a problem. What's one solution? What if we instructed the router to route all incoming traffic directed to a certain port, to a certain computer in our local area network? For example, if three LAN computers share the WAN IP of 1.1.1.1, we can setup a procedure called port forwarding to instruct the router to route all packets arriving on port 6000 to 192.168.1.101:3500.
uPnP is just a fancy technology (not always present in routers - or more importantly, hardly ever enabled by default) that allows you to port forward using code.
If we can't use uPnP to port forward because not all devices have it enabled, there must be another solution - and there is: hole punching.
Hole punching comes in two flavors: UDP and TCP, although TCP hole punching is a bit more complicated because TCP is inherently first a connection-oriented protocol. UDP hole punching is easier because you can send packets without a connection (in fact, is there really such a thing as a connection?)
You can read the procedure on UDP hole punching on Wikipedia and other sites. The concept works like this, though. Any computer sending an outbound packet will expect a reply, meaning the router will allow the inbound packet's reply. At first, two computers will not expect a reply from each other, so the first packet each computer sends will be rejected by the router. If we continue sending packets, however, the router will allow the rest of the packets to come through, because a reply is expected from each computer. As long as this hole-punched connection is maintained active by occasional keep-alive packets, it will stay alive. Unfortunately, this method of UDP hole punching is unsuccessful for Symmetric NATs (explained in the links below). That's why the end-all solution is to simply proxy the connection and traffic through your own server. This requires a farm of computers, so it's quite costly. But this end-all solution is called TURN. STUN is a web service you can query to determine your NAT type and public IP (as opposed to parsing cmyip.com). Just some terms you'll encounter along your personal research.
Playing around with these concepts help. Here are some resources to get your started:
Types of NATs: http://think-like-a-computer.com/2011/09/16/types-of-nat/
A Method for Symmetric NAT Traversal (basically by spamming a lot more packets to a lot more ports): http://www.goto.info.waseda.ac.jp/~wei/file/wei-apan-v10.pdf
Lidgren Networking Library (which has NAT traversal built-in, this is a C# library): http://code.google.com/p/lidgren-network-gen3/w/list