Pthreads - High memory usage

Question

I am programming something in C that creates a lot of Pthreads in Linux on a 256Mb system. I usually have +200Mb free.

When I run the program with a low amount of thread

Answer 1

Yeah, it's fairly common for pretty much any operating system to choke and die on that many threads. It's not a problem that OS makers (Linux or otherwise) have given much attention to since very few systems have that many CPUs, so your code wouldn't likely executing much faster with 100 threads than it would with 8 (or however many CPUs you have).

I think what you need to do is some sort of thread pooling. I also suspect the ulimit answer on this page would be useful if you really need that many threads, but like I said, I don't think that many threads gets you much in most circumstances and if you fix it in your program instead of in the system, it'll make it more portable.