问题
I have a program working on enormous data sets. The objects are best stored on hash implemented containers since the program keeps seeking for objects in the container.
The first idea was to use HashMap since the methods get and remove of this container are more suitable to the uses I need.
But, I came to see the use of HashMap is pretty memory consumable which is a major problem, so i thought switching to HashSet will be better because it only uses <E>, and not <K,V> per element, but when I looked at the implementation i learned it uses an underlying HashMap! this means it wont save any memory!
So this is my questions:
- Are all my assumptions true?
- Is HashMap memory wasteful? more specifically, what is its overhead for each entry?
- Is HashSet just as wasteful as HashMap?
Is there any other Hash based containers which will be significantly less memory consumables?
update
As requested in the comments I will extend a bit on my program, the hashMap is meant to hold a pair of other objects, and some numeric value - a float- calculated from them. along the way it extracts some of them and enters new pairs. Given a pair it needs to ensure it doesnt hold this pair or to remove it. The mapping can be done using the float value or the hashCode of the pair object.
Additionally when i say "enormous data sets" I am talking about ~ 4*10^9 objects
回答1:
There are very useful tips on this site about collections performance in java.
HashSetis built on top of aHashMap< T, Object >, where value is a singleton ‘present’ object. It means thatthe memory consumption of aHashSet is identical to HashMap: in order to storeSIZEvalues, you need 32 * SIZE + 4 * CAPACITY bytes (plus size of your values). It is definitely not a memory-friendly collection.THashSet could be the easiest replacement collection for a
HashSet– it implements Set and Iterable, which means you should just update a single letter in the initialization of your set.
THashSetuses a single object array for its values, so it uses 4 * CAPACITY bytes for storage. As you can see, compared to JDK HashSet, you will save 32 * SIZE bytes in case of the identical load factor, which is a huge improvement.
Also the below image which I took from here can help us keeping something in mind for choosing right collection
回答2:
Are all my assumptions true?
You are correct that HashSet is implemented using HashMap, so you will not save any memory by using HashSet instead.
If you're creating maps with a large number of elements, you should construct your HashMaps with an initialCapacity to the best of your knowledge, in order to prevent repeated rehashing (thus memory thrashing).
Is HashMap memory wasteful? more specifically, what is its overhead for each entry?
No, it's not wasteful. The overhead is an underlying array (size modified by loadFactor), and an Entry object for each key-value pair. In addition to storing a key and value, the entry object also stores a pointer to the next entry in a slot (in case two or more entries are occupying the same slot in the underlying array). The default loadFactor of 0.75 keeps the underlying array size at 133% of the number of entries.
Very specifically, the memory overhead for each entry is:
- the entry object's reference to the key,
- the entry object's reference to the value,
- the entry object's reference to the next entry,
- and the underlying array's reference to the entry (divided by load factor).
It's very difficult to get much more trim than that for a hash-based collection.
Is HashSet just as wasteful as HashMap?
You will gain no memory efficiency by using HashSet instead of HashMap.
Is there any other Hash based containers which will be significantly less memory consumables?
If your keys are primitives (e.g. ints), there are custom Map and Set implementations out there (in third party libraries) which use more memory-efficient data structures.
回答3:
It is true that HashSet uses just as much memory as HashMap. The difference between the two that HasSet implements Set, i.e., it does not care about any value associated with a key, only the presence or lack thereof of a particular value. HashMap is concerned with storing/retrieving (put/get) of values per key.
While HashMap/HashSet store data in an array that is usually slightly larger that the number of elements, this shold not be too much of a problem because the load factor is .75. This means that a HashMap will grow when the number of elements reaches 75% of the size of the underlying array.
A bigger concern than a large map would be lots of empty maps, since the default size of a HashMap is 16. This can be offset by setting the initial capacity to 0.
You can also use TreeMap instead, however, since TreeMap is based on references instead of an array, you will probably waste even more space, especially with larger maps, besides also losing some speed. The main benefit of TreeMap is that it maintains the keys in an ordered state, so if you need them sorted that is the way to go.
Additionally, TreeMap can be used for programming reasons when you either cannot or do not want to make a custom implementation of the equals and hashCode methods of your key type. You can make a comparator for the key type instead. E.g., to make a map/set based on a case-insensitive String, use String.CASE_INSENSITIVE_ORDER as the comparator of a TreeSet
来源：https://stackoverflow.com/questions/28261457/java-hashset-vs-hashmap