python threading memory usage 64 bit vs 32 bit

Question

I have a basic python program that makes a ton of threads (2000), processes something, then writes it out.

I've narrowed down my code to be similar to this (with 2k threads): URL fetch thread example on: http://www.ibm.com/developerworks/aix/library/au-threadingpython/

Except inside my class I literally do nothing (get item from queue, then call task done). Both in this scaled down version, and the version where I do things, memory usage is the same. In 32 bit python interpreter, I use about 105 megs of virtual memory. In 64 bit, I use over 8 gigs.

I'm running rhel 6. I've also added: threading.stack_size(32768) to get stack size down. I'm assuming python is grabbing some default limit for memory to reserve, I just cant figure out what that limit is.

Any ideas?

Thanks!

Answer 1

If you are on RHEL6 or your glibc is newer than 2.10 (you can check with rpm -q glibc). It's due to the missing of MALLOC_ARENA_MAX.

In RHEL 6, the malloc of glibc (>=2.10) has a new arena allocator which allows each thread to be able to allocate its own arena. And the max number of re-usable arena depends on the number of cores. On a 64-bit system, these arenas are 64M mappings and the default number of arenas for a 16-cores system could reach up to 128. You can easily get 128*64MB = 8GB. So it can result in huge amount of virtual memory (VMS) when using many threads (though the increase in RSS could be completely normal.)

This can be resolved by setting the env. variable MALLOC_ARENA_MAX to a small number like 1 or 4.

Answer 2

如果你真的需要2k +线程，你会有兴趣阅读全局解释器锁（GIL）： http ： //wiki.python.org/moin/GlobalInterpreterLock