How to save a very long string in SQlite (sqlite3 python)?

Question

I have a long string that I want to save into a SQlite database (using the sqlite3 module with Python 3.5). It works fine until the string gets to roughly 500,000 characters. If it gets longer than it fails with:

Traceback (most recent call last):
[...]
sqlite3.OperationalError: near "SomeWord": syntax error

My code:

cursor.execute("UPDATE data SET raw_str='{}' WHERE id=1".format(long_string))
connection.commit()

My string is a raw data string of a length of about 6 Mil. Even so, I expect to work with bigger data (10 to 100 times larger).

Answer 1

I can reproduce your error message as follows. Note that long_string ends up with a single-quote character in it.

import json, sqlite3
db = sqlite3.connect('tmp.sqlite3')
cursor = db.cursor()
cursor.execute("create table data (raw_str STRING, id INT)")
long_string = json.dumps(["' SomeWord"])
cursor.execute("Update data SET raw_str='{}' where id=1".format(long_string))

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
sqlite3.OperationalError: near "SomeWord": syntax error

cursor.execute("Update data SET raw_str=? where id=1", [long_string])
# works

You should not use str.format to substitute string data into an SQL query. What seems to have happened is that your data contained a single-quote character, and so you have accidentally carried out an SQL injection attack on yourself.

Lucky for you the string didn't contain '; drop table data; -- '; drop table data; -- '; drop table data; -- .