How many threads/process can i run on my 64bit machine

Question

I want to know : is my system can run 50000 no. of parallel threads/process or not?

For this i changed my 'ulimit max process' and '/proc/sys/kernel/pid_max' to 50000. But still i'm not able to cross ~33000 no. of process/threads.

To count no. of process/threads on my system i am using : ps -eL|wc -l And I wrote a java program to create those no. of threads.

but in last i'm getting this exception:

Java HotSpot(TM) 64-Bit Server VM warning: Attempt to protect stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to protect stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to protect stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Total thread created #**32515**
Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:640)
    at HowManyThreads.main(HowManyThreads.java:12)
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard pages failed.
^CJava HotSpot(TM) 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGINT to handler- the VM may need to be forcibly terminated

Please help me to create 50000 no. of process/thread.

ulimit -a

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 2066250
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 150000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 40000
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

cat /proc/sys/kernel/pid_max

50000

Java program

package create.threads;

public class HowManyThreads
{
    private static Object s = new Object();
    private static int count = 0;
    public static void main(String[] argv)
    {
        try
        {
            for(;;)
            {
                new Thread(new Runnable()
                {
                    public void run()
                    {
                        synchronized(s)
                        {
                            count += 1;
                        }
                        for(;;)
                        {
                            try
                            {
                                Thread.sleep(1000);
                            } catch (Exception e){
                                System.out.println(e);
                            }
                        }
                    }
                }).start();
            }
        }
        finally
        {
            System.out.println("Total thread created #"+count);
        }
    }
}

This is free command output on my system : when my program runs and throws error/exception

free -g
             total       used       free     shared    buffers     cached
Mem:           252          3        248          0          0          0
-/+ buffers/cache:          3        249
Swap:            1          0          1

when i do not run this program

free -g
total       used       free     shared    buffers     cached
Mem:           252          2        250          0          0          0
-/+ buffers/cache:          1        250
Swap:            1          0          1

Could you please help me... where i'm missing something....

Answer 1

Threads use memory on the heap (surprise!)

You can fix your problem by supplying more heap to your Java program:

java -Xmx1024m <whatever comes for your application>

for 1gib of memory.

A rule of thumb was 1mb for a thread, so if you want 50k threads, you need 50gb of memory.

To lower this, you can decrease the stack size per thread

java -Xss512k

for 512k of stack, for your particular example you might get away with only 1k or even less. The default for Linux x64 seems 256k .

BUT I would rather question why you need so many threads to do stuff. You should rather use a threadpool that matches whatever you have in processor cores and then just schedule work to do on this pool.

Answer 2

I believe this is not the number of processes but the memory:

Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:640) at HowManyThreads.main(HowManyThreads.java:12)

The exeption is simply "out of memory" or not?

But there are some more restrictions:

Each process opens stdin/stdout/stderror which means 3 files. 50k Threads means 150K files.

But as you can read here: Maximum number of threads per process in Linux?

the number of threads per process is mostly limited by the size of memory.