I know that transient instance means the instance is newly created and its corresponding row does not exist in the database where as detached instance has corresponding entr
What Matt suggested would only check if the given object is transient or associated with some entity manager.
If you want to check if it's detached or transient (which you shouldn't really care about! it should be transparent) you need to check if the given object has an ID.
if(data.getID() == null) return TRANSIENT;
The ID should only be set for persistent/detached objects. If for some reason you are setting the ID yourself on transient objects then I don't think what you want to do is possible.
If you don't know which field is the ID (for some reason) or you want to make it generic you can try:
ClassMetadata metadata = HibernateUtil.getSessionFactory().getClassMetadata(data.getClass());
if(metadata.getIdentifier(data) == null) return TRANSIENT;