I want a data structure that maps from key to object and vice-versa(unlike HashMaps that map only in a single direction.) An idea could be to store the HashMap within itself
Simplest idea: wrapper class which contains 2 maps, second with swapped keys/values. You will keep O(1) complexity and will use only slightly more memory since you will (probably) keep there reference to object.