redshift / postgresql-如何查找插入期间1秒钟内出现的重复项？

Question

We have a bit of an issue where one of our nodes was logging duplicate events. We use the following query to insert only unique events, however some of the events were fired within 1 second after the previous event so the following query wouldn't catch it as the date field is different.

Can someone help me update this query so it only grabs unique events even if there is 1 second difference?

INSERT INTO project_events
    SELECT * From (
         SELECT 
                session,
                date, 
                team,
                project,
                event_type,
                event_group,
                event_label,
                event_value,
                event_count,

                ROW_NUMBER() OVER ( PARTITION BY 
                    session,
                    date, 
                    team,
                    project,
                    event_type,
                    event_group,
                    event_label,
                    event_value,
                    event_count
                    ORDER BY date, project ASC 
                ) rownum  
         FROM tmp_table_name where record_type='update'
    ) WHERE rownum = 1;

Answer 1

First of all, in your example, putting the same attributes in PARTITION BY and ORDER BY makes little sense, as the values inside each group will be identical, so your query is equivalent to simply doing SELECT DISTINCT on your PARTITION BY attributes

Now, to the real question. How can you know the element is unique? Is it based on the combo of all these attributes: session,team,project,event_type,event_group,event_label,event_value,event_count ?

If so, try this:

SELECT * FROM 
(
    SELECT 
            session,
            date, 
            team,
            project,
            event_type,
            event_group,
            event_label,
            event_value,
            event_count,
            LAG(date) OVER ( 
              PARTITION BY 
                session,
                team,
                project,
                event_type,
                event_group,
                event_label,
                event_value,
                event_count
              ORDER BY
                date
            ) prev_date 
     FROM tmp_table_name where record_type='update'
 ) sub
 WHERE prev_date IS NULL -- first event
    OR DATEDIFF(second, prev_date, date) > 1  -- events more than 1 second apart