Erlang's maximum number of simultaneous open ports?

Question

Does the erlang TCP/IP library have some limitations? I've done some searching but can't find any definitive answers.

I have set the ERL_MAX_PORTS environment variable to 12000 and configured Yaws to use unlimited connections.

I've written a simple client application that connects to an appmod I've written for Yaws and am testing the number of simultaneous connections by launch X number of clients all at the same time.

I find that when I get to about 100 clients, the Yaws server stops accepting more TCP connections and the client errors out with

Error in process  with exit value: {{badmatch,{error,socket_closed_remotely}}

I know there must be a limit to the number of open simultaneous connections, but 100 seems really low. I've looked through all the yaws documentation and have removed any limit on connections.

This is on a 2.16Ghz Intel Core 2 Duo iMac running Snow Leopard.

A quick test on a Vista Machine shows that I get the same problems at about 300 connections.

Is my test unreasonable? I.e. is it silly to open 100+ connections simultaneously to test Yaws' concurrency?

Thanks.

Answer 1

It seems you hit a system limitation, try to increase the max number of open files using

$ ulimit -n 500

Python on Snow Leopard, how to open >255 sockets?

Erlang itself has a limit of 1024:

From http://www.erlang.org/doc/man/erlang.html

The maximum number of ports that can be open at the same time is 1024 by default, but can be configured by the environment variable ERL_MAX_PORTS.

EDIT:

The system call listen() has a parameter backlog which determines how many requests can be queued, please check whether a delay between requests to establish connections helps. This could be your problem.

Answer 2

All Erlang system limits are reported in the Erlang Efficiency Guide:

http://erlang.org/doc/efficiency_guide/advanced.html#id2265856

Reading from the open ports section:

The maximum number of simultaneously open Erlang ports is by default 1024. This limit can be raised up to at most 268435456 at startup (see environment variable ERL_MAX_PORTS in erlang(3)) The maximum limit of 268435456 open ports will at least on a 32-bit architecture be impossible to reach due to memory shortage.

Answer 3

After trying out everybody's suggestion and scouring the Erlang docs, I've come to the conclusion that my problem is with Yaws not being able to keep up with the load.

On the same machine, an Apache Http Components web server (non-blocking I/O) does not have the same problems handling connections at the same thresholds.

Thanks for all your help. I'm going to move on to other erlang based web servers, like Mochiweb.