Performance Counters NextValue() Very Slow (1,000+ Counters)

Question

In our application, we are using the Windows Performance Counters to store some of our application metrics, which are later retrieved in some web services.

I\'m having a

Answer 1

Here’s what I've been able to find out about the counters. Please forgive the grammar; this was somewhat extracted from an email I sent out regarding this problem.

• There is a 4-5 second processing time, on my machine at least (may be better or worse on server, not sure), to read the instance names from a counter category. This varies negligibly with the number of counters in a category. If you are not using instance counters, you can avoid this.
• We store all of the counters in a single category, so it’s inevitable that category will eventually end up with thousands of counters, given our situation. In my testing, the more counters in a category, the worse the performance. This seems like it should make sense, but the performance of an individual counter is affected by the number of counters currently in memory, which is an odd correlation, maybe:
  • With 8 total counters, read time is about 1-2ms per counter
  • With 256 total counters, read time is about 15-18ms per counter
  • With 512 total counters, read time is about 30ms per counter
  • With 3,584 total counters (reading all counters), read time is about 200ms per counter
  • With 3,584 total counters in the system (filtered down in memory, reading only 512 counters), read time is anywhere from 50-90ms per counter. Not sure why these are slower than the previous batch of 512 counters.
  • I ran each of these tests a few times using System.Diagnostics.Stopwatch to time them.
• Of importance to note is the fact that counters have to be read twice because many counters are calculated over a span of time and present an average between start and end read times, so these bad numbers are made worse in a real-world scenario.

Given the numbers above, on my machine, with 512 counters at roughly 50ms each on the slower end, plus the instance query, and the second counter read, we’re looking at about 60 seconds per request. This is given that we’re working with only 512 counters at a time. I’ve run the full query against the service on my machine several times and the request consistently completes in 60-65 seconds.

I certainly would not have assumed this type of performance degradation of single counters based on the number of other counters being assessed. In my reading, the Windows Performance Monitor system is supposed to be fast, and with small collections it certainly is. It’s possible that our use case is not a good fit and we may be abusing the system.

Update

Given that we have control over how we create counters, we have decided to change our approach a bit. Instead of a few categories with many counters, we instead create many categories, each having fewer counters (4-8 counters per category). This approach has allowed us to effectively avoid the performance issue, and counter read times are in the 0-1ms range. In our experience so far, having even 100 new categories with a few counters each does not affect performance in the system at all.

It's important to note when dealing with a large number of additional counters, you will need to address the memory limitation that is set by default for Performance Counters. This can be done either via machine.config or a registry entry. More information can be found here: http://msdn.microsoft.com/en-us/library/ms229387(v=vs.110).aspx