We have an application that is running on 5 (server) nodes (16 cores, 128 GB Memory each) that loads almost 70 GB data on each machine. This application is distributed and serve
Note that while an event handler is subscribed, the publisher of the event holds a reference to the subscriber. This is a common cause of memory leaks in .NET, and in your case it would not be a serious leak but if a managed object is keeping pointer or handle to an unmanaged object then it is not deleting this unmanaged object and so causing memory fragmentation.
If you are sure that the reason for fragmentation is the unmanaged component and that you are not missing something, and if you have access to the code of the umnanaged component you can recompile it and link it using a decent memory allocator like hoard. But this should be done when there is nothing else to do and after serious profiling.