So to understand new/delete better (really to prove to myself with small examples why virtual destructors are needed for interfaces), I want to understand memory leaks, so that
You are causing undefined behaviour. This means that anything can happen. Since something did indeed happen, everything behaves as documented. (Sometimes "something" looks very similar to something else that you might erroneously expect. Doing exactly what you think you were trying to achieve is one of the possible allowed instances of "undefined behaviour".)
Note also that a "memory leak" is sort of the opposite of what you're trying to do - in a memory leak you forget to free memory, whereas you already freed the memory and are now accessing invalid memory.
Here's a real memory leak, which also does not cause undefined behaviour -- don't confuse "bad but correct" with "incorrect" programming!
int * factorial(int * n)
{
if (*n == 0) return new int(1);
else return new int(*n * *factorial(*n - 1));
}