Through my automated crash collection for MaxTo I got the following crash report:
V8.12.0.0 - System.ComponentModel.Win32Exception - :Void UpdateLayered():0
Your program is probably leaking kernel resources. Start diagnosing this problem with Taskmgr.exe. View + Select Columns, check User objects, GDI objects and Handle count. Run your program and observe if any of these is increasing steadily. Once one of them reaches 10,000 your program will die.
With a way to quickly see the leak in action, you can start commenting code to see where the leak occurs. It probably has something to do with your "hook".
Leaking or using to many GDI objects/handles. Those could cause a resource heap shortage. You might not be able to reproduce because your users might have other GDI resource heavy programs running or use Terminal Server in which case they have to share some of the heap with the other users. See System Error. Code: 8. Not enough storage is available to process this command
Here you can read about the Desktop Heap Monitor tool to diagnose desktop heap problems.
Here and here and here are GDI leak detection tools.
I had many custom Windows controls with own resources, so when I create many controls this error is appear. To fix this problem I made Resource file in my library and used outside resources instead of resources on my component code. After that my exception is gone, already tested with 3 times more opened forms and this error gone. So looks like it's a solution.
The problem probably dones't lie in your WndProc - the reason you see it in your call stacks is because pretty much everything GUI-related on Windows go through the WIN32 window procedure. Overriding it in your control simply gives you a hook point to process low-level stuff before higher-level .NET framework processing is done.
This is going to be a complete shot in the dark, but perhaps this post could be relevant? - probably not with those stack traces, though.