SQL JOIN on Temporary Table Reducing Rowcount

Question

I have a table (raw) set up in Amazon Redshift, indexed on a unique ID (VID) and timestamp (timestamp) for each unique data point. Each type of data also has a mode (mode), value (val), and data point id (PID) associated with it.

Timestamp, VID,  PID, VAL, MODE
0001,      V001, P01,   0, DRIVE
0002,      V001, P02,   1, DRIVE

I created a temporary table (_tmp) that has 'start' and 'end' periods for certain events in the data.

MODE, Start,  End, Duration, VID
DRIVE, 0001, 0002,       1,  V001

The temporary table's existence is not related to my question. My concern is also NOT about query time, but instead about rowcounts.

The temporary table has 150 rows in it. I am attempting to get the number of unique timestamps in the original data for a certain PID, where it's value is 0 - within each "start" and "end" of each row of the temporary table.

Essentially -

SELECT count(DISTINCT "r"."timestamp"), "t"."start"
FROM _tmp t
JOIN raw r ON "r"."timestamp" between "t"."start" and "t"."end"
WHERE "r"."pid" = 'P01' AND "r"."VAL" = 0
GROUP BY "t"."start"

The problem I am facing is that, due to (I assume) missing data in the 'raw' table, I am getting a rowcount of ~50 instead of the 150 I need from the _tmp table. Null values are fine, but lessened rows are not. I've tried different types of joins - the issue seems to be that the 'PID' I'm looking for doesn't exist in certain timeranges of the original data.

Answer 1

You are likely correct that the "where" clause is not helping. I think you need (a) a LEFT join to retain all the records in _tmp prior to the filter, and a modified WHERE clause to include the non-matches (which will provide NULL values from those fields).

count() only counts non-null values so it should be fine.

SELECT count(DISTINCT "r"."timestamp"), "t"."start"
FROM _tmp t
LEFT JOIN raw r ON "r"."timestamp" between "t"."start" and "t"."end"
WHERE ( "r"."pid" = 'P01' AND "r"."VAL" = 0 ) OR ("r"."pid" IS NULL AND "r"."VAL" IS NULL)
GROUP BY "t"."start"