SQL-仅保留每天的第一条记录和最后一条记录

Question

I have a table that stores simple log data:

CREATE TABLE chronicle (
    id INT auto_increment PRIMARY KEY, 
    data1 VARCHAR(256),
    data2 VARCHAR(256),
    time DATETIME
);

The table is approaching 1m records, so I'd like to start consolidating data.

I want to be able to take the first and last record of each DISTINCT(data1, data2) each day and delete all the rest.

I know how to just pull in the data and process it in whatever language I want then delete the records with a huge IN (...) query , but it seems like a better alternative would to use SQL directly (am I wrong?)

I have tried several queries, but I'm not very good with SQL beyond JOINs.

Here is what I have so far:

SELECT id, Max(time), Min(time)
FROM   (SELECT id, data1 ,data2, time, Cast(time AS DATE) AS day
        FROM chronicle) AS initial
GROUP BY day;

This gets me the first and last time for each day, but it's not separated out by the data (ie I get the last record of each day, not the last record for each distinct set of data for each day.) Additionally, the id is just for the Min(time).

The information I've found on this particular problem is only for finding the the last record of the day, not each last record for sets of data.

IMPORTANT: I want the first/last record for each DISTINCT(data1, data2) for each day, not just the first/last record for each day in the table. There will be more than 2 records for each day.

Solution: My solution thanks to Jonathan Dahan and Gordon Linoff:

SELECT o.data1, o.data2, o.time FROM chronicle AS o JOIN (
    SELECT Min(id) as id FROM chronicle GROUP BY DATE(time), data1, data2
    UNION SELECT Max(id) as id FROM test_chronicle GROUP BY DATE(time), data1. data2
) AS n ON o.id = n.id;

From here it's a simple matter of referencing the same table to delete rows.

Answer 1

You have the right idea. You just need to join back to get the original information.

SELECT c.*
FROM chronicle c JOIN
     (SELECT date(time) as day, min(time) as mint, max(time) as maxt
      FROM chronicle
      GROUP BY date(time)
     ) cc
     ON c.time IN (cc.mint, cc.maxt);

Note that the join condition doesn't need to include day explicitly because it is part of the time . Of course, you could add date(c.time) = cc.day if you wanted to.

Instead of deleting rows in your original table, I would suggest that you make a new table. Something lie this:

create table ChronicleByDay like chronicle;

insert into ChronicleByDay
    SELECT c.*
    FROM chronicle c JOIN
         (SELECT date(time) as day, min(time) as mint, max(time) as maxt
          FROM chronicle
          GROUP BY date(time)
         ) cc
         ON c.time IN (cc.mint, cc.maxt);

That way, you can have the more detailed information if you ever need it.

Answer 2

this will improve performance when searching on dates.

ALTER TABLE chronicle
ADD INDEX `ix_chronicle_time` (`time` ASC);

This will delete the records:

CREATE TEMPORARY TABLE #tmp_ids (
  `id` INT NOT NULL,
  PRIMARY KEY (`id`)
);

INSERT INTO #tmp_ids (id)
SELECT
    min(id)
FROM
    chronicle
GROUP BY
    CAST(day as DATE),
    data1,
    data2
UNION
SELECT
    Max(id)
FROM
    chronicle
GROUP BY
    CAST(day as DATE),
    data1,
    data2;

DELETE FROM
    chronicle
WHERE
    ID not in (select id FROM #tmp_ids)
    AND date <= '2015-01-01'; -- if you want to consider all dates, then remove this condition