How to scale ejabberd Server machine on CentOS to handle 200 K connections?

Question

I am working on a considerably good ejabberd instance with 40 core CPU machine and 160 GB RAM.

The issue is I am unable to scale up to 200 K parallel connections.

Answer 1

As the first call you would need to establish what's the bottleneck in your case:

• CPU
• Memory
• System limits (open sockets, open files)
• Application architecture

If possible add a resource-tracking application to your node, e.g. recon. It will allow you to check the length of process queues, memory fragmentation, etc. In our production system the amount of memory consumed by Erlang VM was different when reported by the system than when reported by the Erlang VM itself due to Transparent Huge Pages (the system was virtualized). There may be other issues that may not be obvious when inspecting the node using system tools.

So I would propose:

1. Determine processes with the longest queue sizes - they will be responsible for slowing down the system because Erlang VM needs to scan the whole inbox of a process when it receives a message

2. Determine processes with the biggest amount of allocated memory

3. Determine how much memory Erlang itself thinks is allocated

Also, it would be good if you added parameters used to start the Erlang VM.

Addition

Forgot to mention that it may be worth looking at the tuning WhatsApp did to their Erlang nodes to handle hundreds of thousands of simultaneous connections:

The WhatsApp Architecture Facebook Bought For $19 Billion