Python gevent threads not releasing memory

Question

I am using gevent.monkey.patch_all() and then import threading.

From threading I am importing Thread and creating a new thread like that:

thread = Thread(target=handler, args=args)
thread.start()

Then I am executing this code path multiple times and I can see the memory growing and never being released. It may be taking like hours to be release which is not okay as well.

However if I do directly handler(args) I don't get any memory increase.

Do I need to kill the thread some how? Or stop it?

Thanks

Answer 1

If I get you correct, I think you want to put some task at background and execute it periodically or event driven through out the life time. The process which initiate this background task also exist in memory right ?

You can use gevent.spwan to implement this. This is the actual co-routine .

background.py :-

import gevent
import gevent.monkey
import datetime
import time

gevent.monkey.patch_all()


def my_task(*args, **kwargs):
    with open("/tmp/test.log", "a+") as f:
        f.write("time: {}\n".format(datetime.datetime.now()))

def exec_background(task, *args, **kwargs):
    """
    Run Your `task` background with given `args` and `kwargs`.
    """
    def run_periodically(*args, **kwargs):
        while 1:
            gevent.sleep(1)
            task(*args, **kwargs)
    return gevent.spawn(run_periodically, *args, **kwargs)

if __name__ == "__main__":
    exec_background(my_task)
    print "Waiting... Do other stuffs here !"
    time.sleep(100)

This way you can reuse the exec_background method for putting different tasks at background.

$ python background.py

Waiting... Do other stuffs here !

Check the file tail -f /tmp/test.log You can see that contents are appending in background.

Regarding your over memory usage, I think it is mainly related to your task. You might be doing lot of network operations, so make sure that you are cleaning stuffs properly. Python Memory manager does the garbage collection for most of the time, So I don't think any thing wrong with that.

Check your task logic for entire lifetime of your app and chances for accumulating some resources like sockets or some objects. Use Python context managers :).

Hope this will be helpful to you.