How to visualize the behavior of many concurrent multi-stage processes?

Question

Suppose I\'ve got a ton (a continuous stream) of requests to process, and each request has several stages. For example: \"connecting to data source\", \"reading data from data s

Answer 1

I have a multi process software that runs on a machine with 15 cores. Here is what I do.

Log all messages to syslog. Finally plot (selected) last 20 minutes log data on http://www.simile-widgets.org/timeline. To keep an eye on what is getting logged when and the patterns I use syslog viewer. There are plenty you can find the one that suits you. http://www.google.com/search?aq=0&oq=syslog+vi&sourceid=chrome&ie=UTF-8&q=syslog+viewer

Hope this helps.