创建和填充表既缓慢又不稳定

Question

When using csv_copy to create/populate a table, I notice it is extremely slow sometimes. The following are the core code and some sample outputs.

I have two questions:

1. I can't figure out why the time varies for creating and populating tables.
2. I am not sure what caused the "none" to be printed.

Code:

def create_populate_table(table_name,fields,types,cur):
    sql = 'CREATE TABLE IF NOT EXISTS ' + table_name + ' (\n'
    for i in xrange(len(fields)):
        if i==0:
            sql += fields[i]+' '+types[i]+' NOT NULL PRIMARY KEY,\n'
        elif i==len(fields)-1:
            sql += fields[i]+' '+types[i]+')'
        else:
            sql += fields[i]+' '+types[i]+',\n'
    #print sql
    cur.execute(sql)
    conn.commit()
    print "Table ",table_name," created ",timer()

    cur.execute("SELECT count(*) from "+table_name)
    if cur.fetchone()[0]>0:
        return
    # populate data into created table
    fr= open(file, 'r')
    fr.readline()
    # parse and convert data into unicode
    #data = unicode_csv_reader(fr, delimiter='|')
    # anything can be used as a file if it has .read() and .readline() methods
    data = StringIO.StringIO()
    s=''.join(fr.readlines())
    while(s.find('\r\n')<>-1):
        s=s.replace('\r\n','\n')
    #timer()
    while(s.find('||')<>-1 or s.find('|\n')<>-1 ):
        s=s.replace('||','|0|')
        s=s.replace('|\n','|0\n')
    #timer()
    #print s.split('\t')[:2]
    #exit(0)
    data.write(s)
    data.seek(0)
    try:
        cur.copy_from(data, table_name,sep='|')
        conn.commit()
        print "Table ",table_name," populated ",timer()
    except psycopg2.DatabaseError, e:
        if conn:
            conn.rollback()
        print 'Error %s' % e    
    fr.close()  

The outputs I see:

ME_Features_20121001.txt Table ME_Features_20121001 created 1.44s None Table ME_Features_20121001 populated 1.48s None

FM_Features_20121001.txt Table FM_Features_20121001 created 67.92s None Table FM_Features_20121001 populated 0.22s None

NationalFile_20121001.txt (700mb) Table NationalFile_20121001 created 9.34s None Table NationalFile_20121001 populated 4963.18s None

NJ_Features_20121001.txt Table NJ_Features_20121001 created 1.65s None Table NJ_Features_20121001 populated 41.11s None

PW_Features_20121001.txt Table PW_Features_20121001 created 1.73s None Table PW_Features_20121001 populated 0.20s None

Answer 1

How is timer() defined? My blind guess (as you didn't provide its code) is that this function calls print directly to output the measured time, but doesn't return anything explicitly - hence None is printed. If it's still unclear, look at the example below:

>>> def test():
...     print 'test'
... 
>>> print 'This is a', test()
This is a test
None

I'm not sure what you mean saying that the time varies for creating and populating tables . Time needed to populate the table depends on the amount of data to insert, obviously. Time needed to create a table should be more or less the same in each case, so the 67.92s output looks suspicious indeed, but... are you sure it's measured properly?

Again, my blind guess is that timer() prints the time since last call. Perhaps you should explicitly reset it before starting the operation you want to measure? I guess that those 60 seconds were spent before calling create_populate_table() ...