Memory leak happened in Python Google App Engine project. Any efficient way to write my operation?

Question

I have a GAE project written in Python. I made a cron to execute a batch operation. But it hit a soft private memory limit of F1 instance, which is 124MB after a few iterations. Could anyone help me to write this code more efficiently, hopefully within 124MB. len(people) should be less than 500.

def cron():
    q = Account.all().filter('role =', 1)
    people = [e for e in q]
    for p in people:
        s = Schedule.available(p)
        m = ScheduleMapper(s).as_dict()
        memcache.set('key_for_%s' % p.key(), m)

This is dev server and I don't want to upgrade my instance class. Plus, I want to avoid using third party libraries, such as numpy and pandas.

I added a garbage collection in the last line of for loop. But it doesn't seem to be working.

del s
m.clear()
import gc
gc.collect()

Answer 1

To see if it's even possible to fit it into the memory footprint you want modify your query to get a single entity and check if you can execute successfully the for loop for that one entity. Or just add a break at the end of the for loop :)

If that doesn't work you need to upgrade your instance class.

If the experiment works then you can use split the work using Query Cursors into multiple push queue tasks, each processing only one entity or just a few of them.

Maybe take a look at Google appengine: Task queue performance for a discussion about splitting the work in multiple tasks (though the reason for splitting in that case was exceeding the request deadline, not the memory limit).

Note that even when using multiple tasks it's still possible to hit the memory limit (see App Engine Deferred: Tracking Down Memory Leaks ), but at least the work would get done even if a particular instance is restarted (tasks are retried by default).