I had the same problem and the solution was not use
socket.shutdownInput(); socket.shutDownOutput(); until the last time of reading or writing data to the socket. This made the socket go to FIN_WAIT state thus waiting 2 minutes before closing. You can read more about it in this post