Memory leak while inserting new element to database using MySQLdb in python

Question

I have a folder containing a list of images, from which I am creating a hash of each image and inserting it one by one to an sql database using MySQLdb in python. There are around 1400 images in that folder, whose hashes I need to store in that database. However, with each insertion, it is consuming up some portion of my free RAM, which eventually ends up with 0 free RAM space, and the process gets killed. I don't understand why this is happening because one simple insertion should not result in putting the entire database in the RAM. So the amount of RAM that is consumed while doing an insertion should not depend on the size of the database. Yet, my RAM is getting filled up with time, even though I am doing self.db.close() after every insertion. Why is that happening, and how can I fix it?

Here is the code:

def create_hash(userpath, hashfunc=imagehash.phash):
    print "here"
    image_filenames = [os.path.join(userpath, path)
                    for path in os.listdir(userpath) if is_image(path)]
    for img in sorted(image_filenames):
        try:
            hash_img = hashfunc(Image.open(img))
            img_url =  img.split("/")[-1]
            print hash_img
            c = Connection()
            c.insert(img_url, hash_img)
            c.close_connection()
        except Exception,e:
            print str(e)

and here is the insert function of the Connection class:

class Connection():
def __init__(self):
    self.db = MySQLdb.connect("localhost","root","password","some_schema")
    self.cursor = self.db.cursor()

def insert(self, image_url, hash_value):
    query = "insert into image_hash (image_url, hash) value (\"%s\",\"%s\")"%(image_url, hash_value)
    print query
    try:
        self.cursor.execute(query)
        self.db.commit()
    except Exception,e:
        self.db.rollback()
        print str(e)
def close_connection(self):
    self.db.close()

Please note that I am using this imagehash python library in my code above

Answer 1

Try to insert multiple rows in one query.

INSERT statements that use VALUES syntax can insert multiple rows. To do this, include multiple lists of column values, each enclosed within parentheses and separated by commas.

And look is memory still leaking.

class Connection():
    def __init__(self):
        self.db = MySQLdb.connect("localhost","root","password","some_schema")
        self.cursor = self.db.cursor()

    def insert(self, values):
        query = "insert into image_hash (image_url, hash) values {0}".format(values)
        try:
            self.cursor.execute(query)
            self.db.commit()
        except Exception,e:
            self.db.rollback()
            print str(e)

    def close_connection(self):
        self.db.close()


def create_hash(userpath, hashfunc=imagehash.phash):
    _join = os.path.join
    image_filenames = [_join(userpath, path)
                       for path in os.listdir(userpath) if is_image(path)]
    hash_list = []
    for img in sorted(image_filenames):
        try:
            hash_img = hashfunc(Image.open(img))
            img_url =  img.split("/")[-1]
            hash_list.append("('{0}', '{1}')".format(img_url, hash_img))
        except Exception,e:
            print str(e)


    c = Connection()
    c.insert(', '.join(hash_list))
    c.close_connection()