Where does the limitation of the thread pool come from?

Question

Using the following code, I found that the maximum size of the multiprocessing.pool.ThreadPool is 11689 (on my machine). If I make it bigger, I get

RuntimeError: can't start new thread

Can somebody explain where this comes from? 11689 seems like a weird system constant... so maybe I ran out of a resource?

Please note: This is not about how to choose the best number of threads for a Thread pool. This question is about where the 11689 comes from?

Code

Answer 1

A brief look at the CPython source code handling threads suggests the interpreter is not delving into details too much when a thread creation fails. It just raises the RuntimeError you see.

According to pthread_create manpage, a thread creation can fail with the following error codes and reasons.

EAGAIN Insufficient resources to create another thread.

EAGAIN A system-imposed limit on the number of threads was encountered. There are a number of limits that may trigger this error: the RLIMIT_NPROC soft resource limit (set via setrlimit(2)), which limits the number of pro‐ cesses and threads for a real user ID, was reached; the kernel's system-wide limit on the number of processes and threads, /proc/sys/kernel/threads-max, was reached (see proc(5)); or the maximum number of PIDs, /proc/sys/kernel/pid_max, was reached (see proc(5)).

...

My blind guess is that you are incurring in the first problem rather than the second. You might be running out of memory to allocate new threads.

A deeper explanation on ways a thread creation can fail can be found at this link .

This is a blind guess though. A more verbose error handling from the interpreter would surely help.