I have a hostgator VPS hosting account.
I already have a site at my main domain, say mydomain.com
now i want to install node.js server for chat.mydomain.com.
All you need to is to open up a port to listen to requests e.g. if you want to run Node on 8088 and then access it with chat.mydomain.com.
According to their support, you need to contact them to open up 8088 for you > http://support.hostgator.com/articles/pre-sales-policies/open-new-ports