How to increase maximum number of JVM threads (Linux 64bit)

Question

I cannot create more than 32k Java threads in Linux machine with 15G memory.

Answer 1

I ran into a similar problem in a Python program and the following worked for me. This is based on maczniak's answer above and https://superuser.com/questions/1219960/cannot-edit-proc-sys-kernel-threads-max.

echo kernel.threads-max = 1073741823 >> /etc/sysctl.conf && echo 1073741823 > /proc/sys/kernel/threads-max
echo kernel.pid_max = 999999 >> /etc/sysctl.conf && echo 999999 > /proc/sys/kernel/pid_max
echo vm.max_map_count = 2147483646 >> /etc/sysctl.conf && echo 2147483646 > /proc/sys/vm/max_map_count
echo vm.overcommit_memory = 1 >> /etc/sysctl.conf && echo 1 > /proc/sys/vm/overcommit_memory
echo fs.inotify.max_user_instances = 256 >> /etc/sysctl.conf && echo 256 > /proc/sys/fs/inotify/max_user_instances
sysctl -p

I also had to set DefaultTasksMax in /etc/systemd/system.conf (or /etc/systemd/user.conf for user-run services) to DefaultTasksMax=unlimited.

Systemd also applies a limit for programs run from a login-shell. These default to 4096 per user (will be increased to 12288) and are configured as UserTasksMax in the [Login] section of /etc/systemd/logind.conf.

That is from this StackExchange question. Setting my UserTasksMax to UserTasksMax=999999 worked for me.

Answer 2

You can use a sample program to find out the current threads limit.

If you encounter Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread, check these:

1. In small memory machines

   Every Java thread consume its own stack memory. Default stack size is 1024k (= 1M). You can reduce the stack size like java -Xss512k .... JVM cannot be started if the stack size is too low.

   And beware heap memory configurations: (initial) -Xms and (maximum) -Xmx. The more memory is allocated to heap, the less available memory for stack.

2. System limits

   Some values in ulimit -a can affect a thread limit.

   • max memory size - unlimited on most 64bit machines
   • max user processes - linux treats threads like processes
   • virtual memory - unlimited on most 64bit machines. virtual memory usage is increased by -Xss configuration (default 1024k)

   You can change these values by (temporal) running ulimit command or (permanent) editing /etc/security/limits.conf.

3. sys.kernel.threads-max

   This value is the system-global (including non-JVM processes) maximum number of threads. Check cat /proc/sys/kernel/threads-max, and increase if necessary.

   echo 999999 > /proc/sys/kernel/threads-max
   or
   sys.kernel.threads-max = 999999 in /etc/sysctl.conf to change permanently.

4. sys.kernel.pid_max

   If cat /proc/sys/kernel/pid_max is similar to current limit, increase this. Linux treats threads like processes.

   echo 999999 > /proc/sys/kernel/pid_max
   or
   sys.kernel.pid_max = 999999 in /etc/sysctl.conf to change permanently.

   And you may need to increase sys.vm.max_map_count, too.

5. sys.vm.max_map_count

   cat /proc/sys/vm/max_map_count should be at least (2 x thread-count).

   Attempt to protect stack guard pages failed. and OpenJDK 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed. error messages are emitted by JavaThread::create_stack_guard_pages(), and it calls os::guard_memory(). In Linux, this function is mprotect().

   echo 1999999 > /proc/sys/vm/max_map_count
   or
   sys.vm.max_map_count = 1999999 in /etc/sysctl.conf to change permanently.

Answer 3

Additional information for modern (systemd) linux systems.

There are many resources about this of values that may need tweaking (the other answer is a good source for most of them); however a new limit is imposed by way of the systemd "TasksMax" limit which sets pids.max on the cgroup.

For login sessions the UserTasksMax default is 33% of the kernel limit pids_max (usually 12,288) and can be override in /etc/systemd/logind.conf.

For services the DefaultTasksMax default is 15% of the kernel limit pids_max (usually 4,915). You can override it for the service by setting TasksMax in "systemctl edit" or update DefaultTasksMax in /etc/systemd/system.conf