Python多处理分叉的空python进程的私有内存

Question

I'm creating several child processes via Python multiprocessing, but these child processes use a lot heap private memory even when they're empty processes. This situation is getting worse when running on RHEL with THP(Transparent Hugepage) enabled.

• what's in an empty child process's heap private memory?
• Under linux COW (copy-on-write), shouldn't child process shares all memory as it doesn't created/modify any mem pages? or the child process is trying to modify/write mem pages, then what kind of data it's trying to modify/write?
• or it's due to like python object reference count or something?

Here is a simple example to demo this:

import os
import multiprocessing

print parent process's heap memory in /proc/<pid>/smaps

def emptyProcess():
    print child process's heap memory in /proc/<pid>/smaps
    return

multiprocessing.Process(name='p1', target=emptyProcess).start()

Output:

parent:  pid: 20920:   rss:8228864, shr:2781184, priv:5447680, swap:0, pss:6154240

child: pid: 20921:   rss:6397952, shr:5472256, priv:925696, swap:0, pss:3381248

what's in child process's priv memory (925696B, or 664KB in heap)?

Parent process heap memory:

006cc000-00be4000 rw-p 00000000 00:00 0 [heap]

Size: 5216 kB

Rss: 4120 kB

Pss: 4120 kB

Shared_Dirty: 0 kB

Private_Dirty: 4120 kB

Referenced: 4120 kB

Anonymous: 4120 kB

AnonHugePages: 0 kB

KernelPageSize: 4 kB

Child process heap memory:

006cc000-00be4000 rw-p 00000000 00:00 0 [heap]

Size: 5216 kB

Rss: 4396 kB

Pss: 2530 kB

Shared_Dirty: 3732 kB

Private_Dirty: 664 kB

Referenced: 676 kB

Anonymous: 4396 kB

AnonHugePages: 0 kB

KernelPageSize: 4 kB

Answer 1

One of the primary things in each process is the Python interpreter / VM. If this were a C program, you'd see a much different picture, but even with an "empty" Python process, you still incur the overhead of an interpreter unless you use threading. Each Python interpreter has a chunk of heap, stack and code, and Python's multiprocessing is a wrapper (as far as I know) around Linux processes; so basically you are dealing with fork() . Forking a new process means you get a new Python interpreter. Even though the OS is pretty smart about Copy On Write, the overhead of the Python interpreters add up.

My recommendation would be to try Python threads, or switch this to a non-interpreted language to reduce the process overhead.