One thing I am concerned with is that I discovered two ways of registering delegates to events.
You don't need to worry about keeping a reference to the originally registered delegate, and you will not start a "nasty memory pool".
When you call "OnStuff -= new StuffEventHandler(this.Handle);" the removal code does not compare the delegate you are removing by reference: it checks for equality by comparing references to the target method(s) that the delegate will call, and removes the matching delegates from "OnStuff".
By the way, "OnStuff" itself is a delegate object (the event keyword that I assume you have in your declaration simply restricts the accessibility of the delegate).