SQLite finding query-specific index of sub-query

Question

So I know just enough SQL to be dangerous and not much else. I'm developing an application that needs to display sections of data from a SQLite database, and be able to iterate through some of the data based on certain criteria.

So I have a table that looks like this:

create table packets( id INTEGER PRIMARY KEY, timestamp varchar(32), type varchar(32), source varchar(32), destination varchar(32), channel varchar(16), first_sequence integer, last_sequence integer, missing integer );

Then I have a view that will display some or all of the data based on certain criteria.

We're talking about a table with millions of rows, so I can't load it all into memory. So I have a view class that asks for each item individually based upon the index in the view.

So for instance, if I am displaying all of the packets that have a channel value of 'C', and the view is ready to draw the first item I would issue this query to the database:

SELECT * FROM packets WHERE channel='C' LIMIT 1 OFFSET 0

When it's ready to draw the second item I do

SELECT * FROM packets WHERE channel='C' LIMIT 1 OFFSET 1

etc...

This works fine. I realize it's probably not the most optimal way to do it, but I'm just trying to learn how to do it the simplest way with SQLite first and then worry about optimizing later.

Now the problem is I need to be able to find items within this result set based on another search criteria, and get the index within the original result set.

For instance I need to be able to iterate through the items within the original result set that has a "missing" value of 1, and figure out the view index of those items.

If I'm using the entire data set in the table I think I can do it with something similar to this:

SELECT rowid FROM packets WHERE rowid > [currentlySelectedRowID] AND missing=1 LIMIT 1

But when I need an index within a subset of data, the rowid doesn't really help me, and I have no idea how to find the index within the subset without iterating through all of the items individually myself.

Any ideas on how to go about doing this with a SQL query, or pointers to relevant documentation or tutorials would be greatly appreciated.

Thanks!

(note: the question may not make sense as it's a little convoluted in my mind of how to explain it, so if something is not clear I will try to elaborate.)

Edit: actually I found that doing this in C code is sufficient from a performance perspective by querying the entire result set after the initial index by doing a LIMIT -1 OFFSET [startIndex], and then stepping through until I find the next missing item. It would still be nice to know if there is a way to do it with just an SQL statement, though, for future reference.

Answer 1

Note that if a SELECT statement that returns more than one row does not have an ORDER BY clause, the order in which the rows are returned is undefined.

The order appears unimportant to your initial query, but will become important when you need to create subset of the rows (since you want to use the same initial set for the domain of the subset).

Use

SELECT rowid,* FROM packets WHERE channel='C' ORDER BY rowid LIMIT N? OFFSET M?

to impose an order on the results. Then you can do

SELECT rowid 
FROM (SELECT rowid,* FROM packets WHERE channel='C' ORDER BY rowid LIMIT N? OFFSET M?)
WHERE missing=1 LIMIT 1

to find a subset within those.

Addendum , re: does the "rowid" returned from the main SELECT statement reflect the rowid in this temporary table?

Yes...

sqlite> create table packets (channel, missing);
sqlite> insert into packets values ('A',0);
sqlite> insert into packets values ('B',0);
sqlite> insert into packets values ('C',0);
sqlite> select * from packets;
A|0
B|0
C|0
sqlite> create temp table tt as SELECT rowid,* FROM packets WHERE channel='C';
sqlite> select rowid,* from tt;
3|3|C|0
sqlite> insert into packets values ('C',1);
sqlite> drop table tt;
sqlite> create temp table tt as SELECT rowid,* FROM packets WHERE channel='C';
sqlite> select rowid,* from tt;
3|3|C|0
4|4|C|1
sqlite>