Go, tcp too many open files debug

Question

Here\'s a straightforward Go http (tcp) connection test script

func main() {
    ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *htt         


        

Answer 1

If you want to run millions of go routines that open/read/close a socket, well you better up your ulimit, or open/read/close the socket and pass the value read in to the go-routine, but I would use a buffered channel to control how many file descriptors you want to be able to open.

const (
    // this is where you can specify how many maxFileDescriptors
    // you want to allow open
    maxFileDescriptors = 100
)

func main() {
    ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintln(w, "Hello, client")
    }))
    defer ts.Close()
    var wg sync.WaitGroup
    maxChan := make(chan bool, maxFileDescriptors)
    for i := 0; i < 1000; i++ {
        maxChan <- true
        wg.Add(1)
        go func(url string, i int, maxChan chan bool, wg *sync.WaitGroup) {
            defer wg.Done()
            defer func(maxChan chan bool) { <-maxChan }(maxChan)
            resp, err := http.Get(url)
            if err != nil {
                panic(err)
            }
            greeting, err := ioutil.ReadAll(resp.Body)
            if err != nil {
                panic(err)
            }
            err = resp.Body.Close()
            if err != nil {
                panic(err)
            }
            fmt.Printf("%d: %s", i, string(greeting))
        }(ts.URL, i, maxChan, &wg)
    }
    wg.Wait()
}

Answer 2

I had to also to manually set the close connection header to avoid the file descriptor issue:

r, _ := http.NewRequest(http.MethodDelete, url, nil)
r.Close = true
res, err := c.Do(r)
res.Body.Close();

Without r.Close = true and res.Body.Close() I hit the file descriptor limit. With both I could fire off as many as I needed.

Answer 3

HTTP/1.1 uses persistent connections by default:
A significant difference between HTTP/1.1 and earlier versions of HTTP is that persistent connections are the default behavior of any HTTP connection.
http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html
The solution was to inform the server that the client wants to close the connection after the transaction is complete. This can be done by setting the Connection header,
req.Header.Set("Connection", "close") or by setting the Close property to true on the http.Request:
req.Close = true After doing that, the “too many open files” issue went away as the program was no longer keeping HTTP connections open and thus not using up file descriptors.

I solved this by adding req.Close = true and req.Header.Set("Connection", "close"). I think it's better than changing ulimit.

source: http://craigwickesser.com/2015/01/golang-http-to-many-open-files/

Answer 4

I think you need to change your max file descriptors. I have run into the same problem on one of my development VMs before and needed to change the file descriptors max, not anything with inotify settings.

FWIW, your program runs fine on my VM.

·> ulimit -n
120000

But after I run

·> ulimit -n 500
·> ulimit -n
500

I get:

panic: Get http://127.0.0.1:51227: dial tcp 127.0.0.1:51227: socket: too many open files

** Don't fall into the trap that Praveen did **

Note ulimit != ulimit -n.

➜  cmd git:(wip-poop) ✗ ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       1418
-n: file descriptors                4864

Answer 5

Go’s http package doesn’t specify request timeouts by default. You should always include a timeout in your service. What if a client doesn't close their session? Your process will keep alive old sessions hitting ulimits. A bad actor could intentionally open thousands of sessions, DOSing your server. Heavy load services should adjust ulimits as well but timeouts for backstop.

Ensure you specify a timeout:

http.DefaultClient.Timeout = time.Minute * 10

You can validate before and after by monitoring files opened by your process:

lsof -p [PID_ID]

Answer 6

change the ulimit to avoid the error "too many open files" by default max ulimit is 4096 for linux and 1024 for mac, u can change ulimit to 4096 by typing ulimit -n 4096 for beyond 4096 you need to modify limits.conf in etc/security folder for linux and set hard limit to 100000 by adding this line "* hard core 100000"