Fast way to convert array to tuple/list in python?

Question

It seems like the time it takes to convert an array to a tuple scales linearly with the the length of the array. Is there a way to do this more efficient? I need to insert arrays with 5e+6 elements into a mysql database, but MySQLdb only seems to accept tuples or lists as input to insertmany.

Answer 1

MySQLdb uses re and string interpolation to join parameterized SQL with arguments before passing the query to the server as a string. Clearly this is not the way to go -- not only are you converting the array to tuples, but also the tuples into strings.

In contrast oursql sends SQL queries to the MySQL server separate from data.

---

Since you have 5 arrays, using zip (or column_stack ) would require Python (or numpy) to allocate more memory for the combined object (list of tuples or 2D numpy array). To avoid this, use itertools.izip :

import itertools as it
x=np.random.random(1e6)
y=np.random.random(1e6)
connection = oursql.connect(
    host=config.HOST, user=config.USER, passwd=config.PASS, db='test')
with connection.cursor() as cursor:
    sql='INSERT INTO foo (x,y) VALUES (?,?)'
    cursor.executemany(sql,it.izip(x,y))
    print(cursor.lastrowid)

PS. Previously I suggested using oursql.BinaryIterWrapper . I wasn't able to get that solution to work, however, perhaps because of this bug .

PPS. I tried timing the above oursql code versus similar code using MySQLdb. However, no timing was possible for MySQLdb since it raised

_mysql_exceptions.OperationalError: (1153, "Got a packet bigger than 'max_allowed_packet' bytes")

on cursor.ecutemany(...) .