Why is this python script slowly chewing up my RAM?

Question

This script slowly eats my RAM. When I run it, I can see the RAM usage of Python creep up by approx 1mb with each loop, but I can't figure out why. I have figured out that it is the iteration of the query that adds the RAM, but that's all I can figure out. Any help would be awesome.

from haystack.pmod import piliPlacement #this is the SA model
from time import sleep
from datetime import datetime
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker


now = datetime.now()


engine = create_engine('mssql://xxxxxxx;uid=user;pwd=xxxxx',echo=False)


Session = sessionmaker(bind=engine)


def syncPlacements(session):
    query = session.query(piliPlacement).filter(piliPlacement.Time > now)
    pili_placements = [p.ID_Placement for p in query.all()] # this is what adds the RAM
    del pili_placements
    print 'loop'


while True:
    session = Session()
    syncPlacements(session)
    sleep(3)

Answer 1

After stripping it right back, and chatting to a guy on the SA IRC channel, it appeared to be a Mac OSX only problem. So I set it up on Linux but the same thing occured. In the end, I resorted to running the script on a crontab. Works fine now.

M

Answer 2

You might want to see if mdb is slowly getting larger as it does more work. That is why I'd assume that this would be occuring. It is the only object that doesn't look like it would be garbage collected in your example, and it also happens to be an object. So, I would definitely look into what it's doing as it is used more and more.

I'd wonder if it's maintaining a list of queries that have executed.

Answer 3

Just 3 questions

Where does session come from?

Why not put mdb = Connection().haystack into the while True loop (and btw session.close() too)? And pass mdb and session to syncPlacements ?

Whole code block is copied just with just two differences (ie calculate matches, save or remove)?