Python mysql connector prepared INSERT statement cutting off text

Question

I've been trying to insert a large string into an MySQL database using pythons mysql.connector. The problem I'm having is that long strings are getting cut off at some point when using prepared statements. I'm currently using MySQL Connector/Python that is available on MySQL.com. I used the following code do duplicate the problem I'm having.

db = mysql.connector.connect(**creditials)
cursor = db.cursor()

value = []

for x in range(0, 2000):
    value.append(str(x+1))

value = " ".join(value)

cursor.execute("""
                CREATE TABLE IF NOT EXISTS test ( 
                    pid VARCHAR(50),
                    name VARCHAR(120),
                    data LONGTEXT,
                    PRIMARY KEY(pid)
                )  
                """)
db.commit()


#this works as expected
print("Test 1")
cursor.execute("REPLACE INTO test (pid, name, data) VALUES ('try 1', 'Description', '{0}')".format(value))
db.commit()
cursor.close()


#this does not work
print("Test 2")
cursor = db.cursor(prepared=True)
cursor.execute("""REPLACE INTO test (pid, name, data) VALUE (?, ?, ?)""", ('try 2', 'Description2', value))
db.commit()
cursor.close()

Test 1 works as expected and stores all the numbers up to 2000, but test 2 get cut off right after number 65. I would rather use prepared statements than trying to sanitize incoming strings myself. Any help appreciated.

Extra information:

Computer: Windows 7 64 bit

Python: Tried on both python 3.4 and 3.3

MYSQL: 5.6.17 (Came with WAMP)

Library: MySQL Connector/Python

Answer 1

When MySQL Connector driver processes prepared statements, it's using a lower-level binary protocol to communicate values to the server individually. As such, it's telling the server whether the values are INTs or VARCHARs or TEXT, etc. It's not particularly smart about it, and this "behavior" is the result. In this case, it sees that the value is a Python string value and tells MySQL that it's a VARCHAR value. The VARCHAR value has a string length limit that affects the amount of data be sent to the server. What's worse, the interaction between the long value and the limited data type length can yield some strange behavior.

Ultimately, you have a few options:

1. Use a file-link object for your string

   MySQL Connector treats files and file-like objects as BLOBs and TEXTs (depending on whether the file is open in binary or non-binary mode, respectively). You can leverage this to get the behavior you desire.

    import StringIO ... cursor = db.cursor(prepared=True) cursor.execute("""REPLACE INTO test (pid, name, data) VALUES (?, ?, ?)""", ('try 2', 'Description', StringIO.String(value))) cursor.close() db.commit() 

2. Don't use MySQL Connector prepared statements

   If you don't use the prepared=True clause to your cursor creation statement, it will generate full valid SQL statements for each execution. You're not really losing too much by avoiding MySQL prepared statements in this context. You do need to pass your SQL statements in a slightly different form to get proper placeholder sanitization behavior.

    cursor = db.cursor() cursor.execute("""REPLACE INTO test (pid, name, data) VALUES (%s, %s, %s)""", ('try 2', 'Description', value)) cursor.close() db.commit() 

3. Use another MySQL driver

   There are a couple different Python MySQL drivers:

   • MySQLdb
   • oursql