Unable to use more than one processor group for my threads in a C# app

Question

According to MSDN documentation and Stephen Toub answer, my C# app should use every Logical Processor of every Processor Group because it is configured as required (see my

Answer 1

The bug has been fixed by a new (yet unpublished) HP Bios (at the time of writing this).

The new Bios (targeting HP Proliant DL360 and DL380 Gen9) introduce a new setting: "NUMA Group Size Optimization" with choice of [Clustered - default] or [Flat]. HP says to set it to flat.

The sceenshot part of this answer has been conducted on a DL380 instead of a DL360 because of server availability. But I expect same behavior on DL360. The problem disapeared, we had only one group.

As far as I know, the OS communicate with the BIOS to know the CPU(s) configuration. The Bios play an important role in how the OS will present the logical processors available to applications (Processor Group, Affinity, etc).

About the Microsoft documentation Supporting Systems That Have More Than 64 Processors and Processor Groups it is clearly stated that more than one processor group will only be created when the Logical Processor (LC) count is >64. On our server (56 LC) with Numa Architecture set to "Clustered" we had 2 processor groups. A hardware engineer working at HP Bios dev team explained me that when set to "Clustered", the Bios is fooling Windows by padding the real number of logical processor to 72 Logical Processor (the max number of Logical Processor for the E5 v3 Family). The real number of LC is 56 in our DL360. That's the reason why we add 2 groups instead of 1. The Microsoft documentation seems accurate. I personally think that it would be better to create 1 group per numa node whenever possible but in our case, there is a bug. What is faulty is hard to know between HP or Microsoft when the HP Bios setting is set to Clustered (default) but Microsoft seems to not support that option which seems to cause our problem.

On HP Bios for DL360 and DL380, The Bios configuration "Numa Configuration" set to "Clustered" (default) will create 2 groups although there is only 56 Logical Processors (when hyperthreaded). The result is that only one processor is visible at a time for any application. Probably also due to HP fooling Windows by padding fake number of Logical Processors. It sounds like Microsoft does not expect that. Our C# app can't run on the 2 groups. It's hard to blame Microsoft on that behavior where HP does something they can't anticipated. Perhaps we will see, one day, Windows supporting many groups when LC <= 64.

About Prime95. This CPU stress test software has good documentation on Wikipedia that clearly state that it will load into only one processor group (in Limits section).

Answer 2

Try setting your code to build "optimize code" and the target platform to "x64". (it worked for me with your code, on a server with 80 cores)

This is our MsInfo32:

OS Name Microsoft Windows Server 2012 R2 Standard

Version 6.3.9600 Build 9600

Other OS Description Not Available

OS Manufacturer Microsoft Corporation

System Manufacturer IBM

System Model System x3850 X5

System Type x64-based PC

System SKU

Processor Intel(R) Xeon(R) CPU E7- 4870 @ 2.40GHz, 2395 Mhz, 10 Core(s), 20 Logical Processor(s)

Processor Intel(R) Xeon(R) CPU E7- 4870 @ 2.40GHz, 2395 Mhz, 10 Core(s), 20 Logical Processor(s)

Processor Intel(R) Xeon(R) CPU E7- 4870 @ 2.40GHz, 2395 Mhz, 10 Core(s), 20 Logical Processor(s)

Processor Intel(R) Xeon(R) CPU E7- 4870 @ 2.40GHz, 2395 Mhz, 10 Core(s), 20 Logical Processor(s)

BIOS Version/Date IBM Corp. -[G0E179BUS-1.79]-, 28-07-2013

SMBIOS Version 2.7

Embedded Controller Version 255.255

BIOS Mode UEFI

BaseBoard Manufacturer IBM

BaseBoard Model Not Available

BaseBoard Name Base Board

Platform Role Enterprise Server

Secure Boot State Unsupported

PCR7 Configuration Not Available

Hardware Abstraction Layer Version = "6.3.9600.17031"

User Name Not Available

Time Zone Romance Standard Time

Installed Physical Memory (RAM) 128 GB

Total Physical Memory 128 GB

Available Physical Memory 53,0 GB

Total Virtual Memory 147 GB

Available Virtual Memory 67,7 GB

Hyper-V - VM Monitor Mode Extensions Yes

Hyper-V - Second Level Address Translation Extensions Yes

Hyper-V - Virtualization Enabled in Firmware Yes

Hyper-V - Data Execution Protection Yes

Answer 3

And you should ensure there is no Job restrictions.

https://msdn.microsoft.com/en-us/library/windows/desktop/ms684147(v=vs.85).aspx

You can check by Process Explorer

https://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

Answer 4

Will this program deadlock or not? I can't determine your thread pool is fully expanded or not.

using System.Linq;
using System.Threading;
using System.Threading.Tasks;

class Program
{
    static void Main(string[] args)
    {
        var threads = 100;
        int workerThreads, completionPortThreads;

        ThreadPool.GetMaxThreads(out workerThreads, out completionPortThreads);
        ThreadPool.SetMaxThreads(threads, completionPortThreads);
        ThreadPool.GetMinThreads(out workerThreads, out completionPortThreads);
        ThreadPool.SetMinThreads(threads, completionPortThreads);

        var ce = new CountdownEvent(threads + 1);
        Enumerable.
            Range(0, threads).
            Select(xs => Task.Factory.StartNew(() => { ce.Signal(); ce.Wait(); })).
            ToArray();

        ce.Signal();
        ce.Wait();
    }
}