golang too many open files in go function, goroutine [closed]

问题

package main

import (
    "os"
    "sync"
)

func main() {
    var wg sync.WaitGroup
    wg.Add(1024 * 1024)
    for i := 0; i < (1024 * 1024); i++ {
        go func(index int) {
            if f, e := os.Open(i); e == nil {
                //blah blah
                f.Close()
            }
        }(i)
    }
    wg.Done()
}

If you run the program brings up the following error. "open $ too many open files" Please tell us how to eliminate the error.

回答1:

You are running out of system resources because you are using up too many file descriptors without releasing enough of them. You need to limit concurrency in your program.

For this, you can have a buffered channel that acts as a counting semaphore.

sem := make(chan struct{}, 12) // 12 is the maximum number of 
                                // concurrent processes that may run at any time

Now you can modify your method as:

func main() {
    var wg sync.WaitGroup
    wg.Add(1024 * 1024)

    for i := 0; i < (1024 * 1024); i++ {
        go func(index int) {

            // if there are already 12 goroutines running, below send will block
            // and a new file wont be open
            sem <- struct{}{}

            // once this goroutine finishes, empty the buffer by one
            // so the next process may start (another goroutine blocked on
            // above send will now be able to execute the statement and continue)
            defer func() { <-sem }()

            // wg.Done must be deferred after a read from sem so that
            // it executes before the above read
            defer wg.Done()

            if f, e := os.Open(strconv.Itoa(index)); e != nil {
                // handle file open failure
                return
            }
            defer f.Close()
            // handle open file
        }(i)
    }

    wg.Wait()
    close(sem)
}

Your use of wg.Done is also incorrect. Read about it here

(Note that this code is to give a basic idea about this kind of problem. You can also refer to this question for a working example: Go worker pool with repetitive queue structure

来源：https://stackoverflow.com/questions/38824899/golang-too-many-open-files-in-go-function-goroutine

标签

goroutine