Why does function take a lot longer when passed an element from a large vector than a small vector?

Question

I have some code (shown at the end of this post) that essentially just loops over a vector of pointers, reading the current element and then calls a function, fitFcn, calcul