Bckground
I have a networked application written in C#. my server program has a UI and several communication threads, that read from tcp sockets and display
You are calling BeginInvoke() too often. The UI thread will dispatch the delegate before it does its normal duties, like painting controls and responding to the mouse. If the delegate target is doing a lot of work (appending text to an RTB is expensive) or you are simply flooding it with requests, it doesn't get around to doing its normal job anymore. The UI will freeze. That happens pretty easily, about a 1000 invokes per second is enough.
The key to solve this is to realize that it is wasted effort to invoke so often. The human eye can't perceive updates faster than 25 times per second. So buffer the text until 50 milliseconds have passed since the last update. Environment.TickCount is a cheap timer.
Also beware that you might actually produce text faster than can be appended to the RTB. That requires throttling the thread. That's easy: use Invoke instead of BeginInvoke.