How to run the maximum number of goroutines

问题

The code is like this:

func find(start int, end int){
    for i := start; i < end ; i++ {
        // Do something...
    }
}

func main() {
    // Do something...
    // For example, I know: len = 1500
    // and run max goroutine are 3
    // will be
    go find(0, 500)
    go find(500, 1000)
    go find(1000, 1500)
    // Do something...
}

That is when I know in advance the maximum threads of goroutines and the length of "length". But if I do not know how many threads can run at goroutine, and the length of "length". Is there any way to divide "length" into equal sections for thread processing? For example: length = 10, max goroutine that can run is 2 and it will split length into 2 threads (10/2, each length is 5) to be able to handle simultaneously.

回答1:

Maximizing throughput is about getting rid of bottlenecks. First of all find where is most time is lost.

Sometimes running too many goroutines doesn’t make things faster but makes them slower, because goroutines start to compete for resources. Usually the fastest configuration is one that uses resources in the most effective way: if things are limited by computation then it's best to set the number of running goroutines equal to the number of CPU cores. But if it is limited by IO then choose the optimal number by measuring its performance.

Find the main thing that is slowing down your program - then make it faster.

回答2:

I know I've seen and read references on job scheduling strategies, but I can't find a good one right now. But you've got some basic options:

Split your work up ahead of time into equal chunks.
Pick a chunk size and issue chunks to threads until done.
Work stealing: create threads, issue chunks, threads that finish early steal work from other threads.

All of these work in Go. To find the number of real threads the Go runtime has available, look at https://golang.org/pkg/runtime/#GOMAXPROCS

Also, just from experience, channels are tempting. But for best performance don't send single work items in a channel. Send array / slices of work items in each message instead. Otherwise the channel's locking overhead can become more time consuming than the actual work.

回答3:

The best way I can think of would be a fixed size worker pool of goroutines, as described in https://gobyexample.com/worker-pools

来源：https://stackoverflow.com/questions/47113585/how-to-run-the-maximum-number-of-goroutines

标签

multithreading

max

goroutine