Execute sqlite3 “dot” commands from Python or register collation in command line utility

Question

My sqlite3 database contains a "collate" column-constraint. I've placed it in the schema for the table, to prevent accidentally neglecting to use the necessary collation. However this means when running sqlite3 from the command line, and not from my Python code, the collation referenced in the schema is not present, and I'm unable to use dot commands.

sqlite> .import data.txt table_name
Error: no such collation sequence: my_collation

Furthermore, creating the connection from Python, and adding the collation required runs into this problem:

connWithCollation.execute(".import data.txt table_name")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
sqlite3.OperationalError: near ".": syntax error

The execute function it would appear does not want to pass the sqlite3 dot command through.

How can I execute sqlite3 dot commands when necessary collation functions are not present? Alternatively, how can I execute sqlite3 dot commands from Python?

Answer 1

You can call dot commands from Python using the subprocess module, which basically invokes a shell. If you need to use multiple dot commands, you can pass them as separate shell arguments - using a semicolon to separate them won't work.

import subprocess
subprocess.call(["sqlite3", "xxx.db", 
  ".mode tabs", 
  ".import file.tsv table_name"])

Answer 2

You can load new collating sequences and functions using load_extension() built-in SQLite function or .load command in command line shell for SQLite. Obviously, extensions shold be written in C.

And you can not call dot-commands from python, because dot-commands are specific to the command line shell tool.

Answer 3

The .import command in the sqlite shell is a builtin command. It's processed by the shell program, not the SQL engine, so you can't execute it like an SQL statement.

Reading code for SQLite's shell.c , it seems that .import is simply a loop, reading lines from the data file, splitting on the separator, and passing the fields as parameter values to a prepared INSERT statement. So you should be able to mimic the behavior of .import with Python code easily.

I tested the following with Python 2.6:

import sqlite3
import csv

conn = sqlite3.connect(':memory:')

conn.execute('create table mytable (col1 text, col2 text, col3 text)')

csvReader = csv.reader(open('mydata.csv'), delimiter=',', quotechar='"')

for row in csvReader:
        conn.execute('insert into mytable (col1, col2, col3) values (?, ?, ?)', row)

cur = conn.cursor()
cur.execute('select * from mytable')
print cur.fetchall()

Answer 4

The dot commands are only available to the sqlite3 executable. You will have to replace them with the equivalent combination of Python code and DB-API calls in order to mimic their behavior.

Answer 5

The APSW Python wrapper for SQLite does include a shell compatible with the SQLite one. You can also extend it to add more commands of your own choosing, extra output modes etc. (Disclosure: I am the APSW author.)

• APSW shell
• Difference between APSW and pysqlite

Answer 6

subprocess.run() is preferred over subprocess.call() . Below is my answer:

import subprocess
schema = subprocess.run(
    ['sqlite3',
     'xxx.db',
     '.schema'
    ],
    capture_output=True
).output.decode('utf-8')

print(schema)