You can do the calculation based on an example. Suppose your array passed to replace() has 10 elements (n = 10). So, the loop inside of replace() runs 10 times, and will call findMax 10 times. Because the loop inside findMax will only start running from i+1, the exact number of times it will run, based on the ten method calls, is (9-i) times:
outer i | inner loop | number of loops
=======================================
0 | from 1 to 9 | 9
1 | from 2 to 9 | 8
... | ... | ...
8 | from 9 to 9 | 1
9 | no loop | 0
So your formula for the number of inner loops is 9 + 8 + 7 + ... + 1 = 45. That is roughly 1/2 n², and cleary more than n (which we assumed to be 10 in this example). As the big-o notation will disregard constant values, we can leave the 1/2 away, and simply say that the complexity is O(n²).