脚本优化，可进行大量更新

Question

I'm working on a script which creates a hash of some data and save it in the database. The data needed come from a SQL query which join around 300k rows with 500k rows. While parsing results, I create the hash value and update in the db using a second connexion handler (using the first one gives me a "Unread results" error).

After a lot of investigations, I figured out that what give me the best results in terms of performances are the following:

• Restart the select query every x iterations. Otherwise, updates become a lot slower after a certain time
• Commit only every 200 queries instead of committing on every query
• Tables used for the select query are MyISAM and indexed with primary key and fields used in join.
• My hash table is InnoDB and there's only the primary key (id) which is indexed.

Here's my script:

commit = ''       
stillgoing = True    
limit1 = 0
limit2 = 50000    
i = 0    
while stillgoing:    
    j = 0    
    # rerun select query every 50000 results
    getProductsQuery = ("SELECT distinct(p.id), p.desc, p.macode, p.manuf, "
        "u.unit, p.weight, p.number, att1.attr as attribute1, p.vcode, att2.attr as attribute2 "
        "FROM p "
        "LEFT JOIN att1 on p.id = att1.attid and att1.attrkey = 'PARAM' "
        "LEFT JOIN att2 on p.id = att2.attid and att2.attrkey = 'NODE' "
        "LEFT JOIN u on p.id = u.umid and u.lang = 'EN' "
        "limit "+str(limit1)+", "+str(limit2))                           
    db.query(getProductsQuery)
    row = db.fetchone()              
    while row is not None:
        i += 1
        j += 1
        id = str(row[0])
        # create hash value
        to_hash = '.'.join( [ helper.tostr(s) for s in row[1:]] )
        hash = hashlib.md5(to_hash.encode('utf-8')).hexdigest()
        # set query
        updQuery = ("update hashtable set hash='"+hash+"' where id="+id+" limit 1" )         
        # commit every 200 queries
        commit = 'no'
        if (i%200==0):
            i = 0
            commit = 'yes'
        # db2 is a second instance of db connexion
        # home made db connexion class
        # query function takes two parameters: query, boolean for commit
        db2.query(updQuery,commit)            
        row = db.fetchone()        
    if commit == 'no':
        db2.cnx.commit()            
    if j < limit2:
        stillgoing = False
    else:
        limit1 += limit2

Currently the script takes between 1 hour 30 and 2 hours to run completely. These are the better performances I got since the very first version of the script. Is there anything I can do to make it run faster?

Answer 1

I think you should be able to do this entirely within MySQL:

updateProductsQuery = "
    UPDATE hashtable AS h
    JOIN p ON h.id = p.id
    LEFT JOIN att1 on p.id = att1.attid and att1.attrkey = 'PARAM'
    LEFT JOIN att2 on p.id = att2.attid and att2.attrkey = 'NODE' 
    LEFT JOIN u on p.id = u.umid and u.lang = 'EN'
    SET h.hash = MD5(CONCAT_WS('.', p.desc, p.macode, p.manuf, u.unit, p.weight, p.number, att1.attr, p.vcode, att2.attr))
    LIMIT " + str(limit1) + ", " + str(limit2)

Answer 2

... LIMIT 0,200  -- touches 200 rows
... LIMIT 200,200  -- touches 400 rows
... LIMIT 400,200  -- touches 600 rows
... LIMIT 600,200  -- touches 800 rows
...

Get the picture? LIMIT + OFFSET is O(N*N). Quadraticly slow.

To get it down to O(N), you need to do a single linear scan. If the single query (with no LIMIT/OFFSET), takes too long, then walk though the table in 'chunks':

... WHERE id BETWEEN 1 AND 200  -- 200 rows
... WHERE id BETWEEN 201 AND 400  -- 200 rows
... WHERE id BETWEEN 401 AND 600  -- 200 rows
... WHERE id BETWEEN 601 AND 800  -- 200 rows

I blog about such here . If the table you are updating is InnoDB and has PRIMARY KEY(id) , then chunking by id is very efficient.

You could have autocommit=1 so that each 200-row UPDATE automatically COMMITs .

Oh, your table are using the antique Engine, MyISAM? Well, it will run reasonably well.