How to limit memory and CPU usage in Python under Windows?

Question

I write deep learning software using Python and the Tensorflow library under Windows. Sometimes by mistake I load too much into memory and the computer stops responding; i cannot even kill the process.

Is it possible to limit the memory and CPU usage for Python scripts under Windows? I use PyCharm as an editor. Under UNIX Systems there seems to be the possibility to use resource.RLIMIT_VMEM , but under Windows I get the notification no module named resource .

Answer 1

You can, of course, use the Win32 Jobs API (CreateJobObject & AssignProcessToJobObject) to spawn your program as a sub-process and manage its resources.

But I guess a simpler solution, without going through all the hassle of coding, is to use Docker to create a managed environment.

Answer 2

This is a common problem when running resource-intensive processes, where the total amount of memory required might be hard to predict.

If the main issue is the whole system halting, you can create a watchdog process preventing that from happening and killing the process. It is a bit hacky, not as clean as the UNIX solution, and it will cost you a bit of overhead, but at least it can save you a restart!

This can easily be done in python, using the psutil package. This short piece of code runs whenever over 90% of virtual memory has been used and kills the python.exe process which is using the most memory:

import time
import psutil
while True:
    if psutil.virtual_memory().percent > 90:
        processes = []
        for proc in psutil.process_iter():
            if proc.name() == 'python.exe':
                processes.append((proc, proc.memory_percent()))
        sorted(processes, key=lambda x: x[1])[-1][0].kill()
    time.sleep(10)

This can also be adapted for CPU, using psutil.cpu_percent() .

Answer 3

It's a bad practice to limit the CPU usage. You're doing something wrong, fix it instead of working around it.

You said:

Sometimes by mistake I load too much into memory and the computer stops responding;

What do you exactly mean by mistake ? You should learn from your mistakes.

I'd suggest running the heavy processing part in a separate thread so your script wont stop responding.