sqlite3 datetime.datetime python select

Question

The sqlite3 database is hundreds of thousands of rows. I want to sort it by the column with (datetime.datetime(now))

The table might be:

c.execute(“CREATE TABLE IF NOT EXISTS table_one (time_column TEXT,column_two REAL,column_three REAL)”)

The time column is TEXT

The inserts have worked fine and I have hundreds of thousands of rows. I need to look at the data by time. For example “what is the average and standard deviation on January 3, 2019 between 1:00am and 2:00am” (this would be about 300 numbers)

Something like this:

all_rows =  c.execute('SELECT  column_two   FROM table_one WHERE time_column >  2019-01-03  01:00:00:000000  and time_column <  2019-01-03  02:00:00:000000  ‘)

The fetchall needs to go into an array for statistical analysis, but that is another problem.

At present the retrieve of the datetime is fine. It comes across as a unicode (whatever that is)

But I need to make decisions with the datetime field. Is the TEXT a problem? I could add a column and change the date stamp to a 'unix time stamp' which appears to be a strait number.

I am learning more and more, but stuck on this …. help (and thanks)

Answer 1

You are essentially asking for advices on how to design a database to improve efficiencies of some requests. That could be rather broad but I will try for this simple example.

First (and for your question), the TEXT type for the time column is acceptable for SQLite which has no dedicated Date type. The date are converted in ISO8601 strings ("YYYY-MM-DD HH:MM:SS.SSS"), which ensures compatibility with the comparisons operators. The good questions to ask is what is the range for your date values, and whether you need to evaluate date differences. If you only need a one second precision with dates greater that 1970-01-01, then INTEGER (number of seconds since 1970-01-01 00:00:00 UTC) is good, and REAL (Julian day numbers) is good if exact precision does not matter (because of floating point inaccuracy), but a large range is required.

But what matters most for query efficiency is the presence of an index. So if you need to improve queries using the time_column , declare an index on it. Your code could become:

c.execute("CREATE TABLE IF NOT EXISTS table_one (time_column TEXT,column_two REAL,"
           "column_three REAL)")
c.execute("CREATE INDEX IF NOT EXISTS index_time_table_one ON table_one(time_column)")

and the select part would be:

curs = c.execute("SELECT  column_two   FROM table_one WHERE time_column"
                 "BETWEEN '2019-01-03 01:00:00'  and '2019-01-03 02:00:00'")

Simply that way, you will get an ISO string representations when you fetch the time_column column.

Python SQLite3 module is even smarter with dates, and is able to automatically convert them to timestamp, provided you kindly ask it:

# declare that you want to use custom datatypes, declare in columns
c = sqlite3.connect('your_db', detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)
# declare the time_column to be a timestamp:
c.execute("CREATE TABLE IF NOT EXISTS table_one (time_column timestamp,column_two REAL,"
           "column_three REAL)")
# declare the index
c.execute("CREATE INDEX IF NOT EXISTS index_time_table_one ON table_one(time_column)")

When you database is populated you can fetch it as usual and retrieve directly Python datetime object:

curs = c.execute("SELECT * FROM table_one WHERE time_column BETWEEN"
                 " '2019-01-03 01:00:00'  and '2019-01-03 02:00:00'")
r = curs.fetchone()
print(r)

will output:

(datetime.datetime(2019, 1, 2, 0, 0), ...)

The drawback is that is is no longer standard SQL and will require tweaking if you use a different database.