Say an ArrayList is of size n.
In my case, I often need to remove from 1 to n elements with different indexes from an ArrayList.
By using visualvm profiler,
Check out the list of datastructures here. Pick one depending on your requirements. Like Guarev mentioned, a HashMap is probably your best bet. Hashmaps have the advantage of a constant time for insert, search, and delete.
ArrayLists are not a good structure for a storing a lot of data, as the memory usage quickly goes through the roof, and search/delete times get very expensive very quickly.