Java threading optimization at 100% CPU usage

Question

I have an application that accepts work on a queue and then spins that work off to be completed on independent threads. The number of threads is not massive, say up to 100, but

Answer 1

The fact that your CPU is running at 100% does not tell much about how busy they are doing useful work. In your case, you are using more threads than cores so the 100% includes some context switching and uses memory unnecessarily (small impact for 100 threads), which is sub-optimal.

For CPU intensive task, I generally use this idiom:

private final int NUM_THREADS = Runtime.getRuntime().availableProcessors() + 1;
private final ExecutorService executor = Executors.newFixedThreadPool(NUM_THREADS);

Using more threads, as others have indicated, only introduces unnecessary context switching.

Obviously if the tasks do some I/O and other blocking operations, this is not applicable and a larger pool would make sense.

EDIT

To reply to @MartinJames comment, I have run a (simplistic) benchmark - result shows that going from a pool size = number of processors + 1 to 100 degrades the performance only slightly (let's call it 5%) - going to higher figures (1000 and 10000) does hit the performance significantly.

Results are the average of 10 runs:
Pool size: 9: 238 ms. //(NUM_CORES+1)
Pool size: 100: 245 ms.
Pool size: 1000: 319 ms.
Pool size: 10000: 2482 ms.

Code:

public class Test {

    private final static int NUM_CORES = Runtime.getRuntime().availableProcessors();
    private static long count;
    private static Runnable r = new Runnable() {

        @Override
        public void run() {
            int count = 0;
            for (int i = 0; i < 100_000; i++) {
                count += i;
            }
            Test.count += count;
        }
    };

    public static void main(String[] args) throws Exception {
        //warmup
        runWith(10);

        //test
        runWith(NUM_CORES + 1);
        runWith(100);
        runWith(1000);
        runWith(10000);
    }

    private static void runWith(int poolSize) throws InterruptedException {
        long average = 0;
        for (int run = 0; run < 10; run++) { //run 10 times and take the average
            Test.count = 0;
            ExecutorService executor = Executors.newFixedThreadPool(poolSize);
            long start = System.nanoTime();
            for (int i = 0; i < 50000; i++) {
                executor.submit(r);
            }
            executor.shutdown();
            executor.awaitTermination(10, TimeUnit.SECONDS);
            long end = System.nanoTime();
            average += ((end - start) / 1000000);
            System.gc();
        }
        System.out.println("Pool size: " + poolSize + ": " + average / 10 + " ms.  ");
    }
}