Python's Sqlite3 module will not return full rows when using SELECT FROM WHERE

Question

I'm trying to create a function using Python's Sqlite3 module that will return a list of rows based on a datetime timestamp search using the SELECT command.

Right now, selecting everything (with 'select * from example_table' ) will return a entire rows correctly, but selecting based on timestamps (using '''select %s from %s where %s > ? and %s < ?''' % (date_col_name, table_name, date_col_name, date_col_name) ) will only return the timestamp in a tuple (missing the other column).

Previously, I struggled to preserve the datetime data type, but with detect_types=sqlite3.PARSE_DECLTYPES it returns an actual datetime object. Normally I would suspect this to be the issue, however the actual "filtering" part of the SELECT command is working. The correct datetime objects are being returned, they're just missing the other datas in the row they belonged to.

Relevant code:

sql_database_name = 'data_history.db'
date_col_name = 'Date'

class Database manager ...

    ... init ...

    def get_table_range(self, table_name, daterange=None):
        con = sqlite3.connect(self.database_name, detect_types=sqlite3.PARSE_DECLTYPES)
        c = con.cursor()

        if daterange is not None:
            startdate = daterange[0]
            enddate = daterange[1]
            sql = '''select %s from %s where %s > ? and %s < ?''' % (date_col_name, table_name, date_col_name, date_col_name)
            data = (startdate, enddate)
            c.execute(sql, data)
        else:
            sql = 'select * from %s' % table_name
            c.execute(sql)

        print("Fetchine one: ", c.fetchone())

        result = c.fetchall()

        c.close()
        con.close()

        return result


if __name__ == "__main__":
    test_db = "test.db"
    manager = DatabaseManager(database_name=test_db)

    selected = manager.get_table_range("test_table")
    print("Selected " + str(len(selected)) + "rows.")

    print("---")

    selected = manager.get_table_range("test_table", (datetime(2020, 3, 2, 23), datetime(2020, 3, 3)))
    print("Selected " + str(len(selected)) + "rows.")

Actual output:

Fetchine one:  (datetime.datetime(2020, 3, 2, 19, 12, 57, 120184), 291.0)
Selected 97rows.
Fetchine one:  (datetime.datetime(2020, 3, 2, 23, 22, 15, 704786),) <<-- extra columns were not returned
Selected 25rows.

Desired output:

Fetchine one:  (datetime.datetime(2020, 3, 2, 19, 12, 57, 120184), 291.0)
Selected 97rows.
Fetchine one:  (datetime.datetime(2020, 3, 2, 23, 22, 15, 704786), XXX.X) 
Selected 25rows.

Answer 1

It's returning exactly what you're asking for, since your query translates into:

 SELECT date_col_name FROM table_name WHERE date_col_name < ? AND date_col_name > ?;

You probably want to generate the code SELECT * FROM. . . SELECT * FROM. . . or SELECT list, of, column, names FROM . . . SELECT list, of, column, names FROM . . . .

And why are you building the SQL statement with string replacement for the column and table names? You should just be writing:

 sql = '''select date_col_name from table_name where date_col_name > ? and date_col_name < ?''' 

The column names are not variables (unless you have some strange set-up with multiple identically structured tables with different name and different column names, which could possibly indicate a design flaw).