The simple answer: do not mix arrays with generics!
Use a list of hashsets:
ArrayList> rows = new ArrayList>();
The problem here is that Java specification doesn't allow you to declare an array of generics object.
A workaround of it is to use the wildcard, but you will lose the type-safety:
HashSet>[] rows = new HashSet>[9];
for (int i = 0; i < rows.length; ++i)
rows[i] = new HashSet();
This, in your case, won't create problems when you are going to check if an item is contained: you can easily do rows[0].contains(4)
but when adding new content you will be forced to cast the row to the right type and suppress the warning of unchecked cast itself:
((HashSet)rows[0]).add(4);
A side note: if you feel pioneer just download the Trove Collections framework that has a not-generics, highly optimized version of an integer hashset made to work with primitive type, I'm talking about the TIntHashSet class: it will solve your problem and you'll end up having faster code.