I hate EventHandler. I hate that I have to cast the sender
if I want to do anything with it. I hate that I have to make a new class inheriting from EventArgs<
Well, doing all that casting is unusual, the client code quite often already knows who the sender is because it explicitly subscribed the event for only one object. Sharing event handlers is fairly rare. If common behavior is desirable then the better approach is to derive from the class and override the OnXxxx method. You then no longer care about sender, you got this.
But, solve your problem trivially by including a type-safe reference to the sender in your custom EventArgs derived class.