MySQL，返回X小时内的所有测量和结果

Question

This question is very much related to my previous question: MySQL, return all results within X last hours altough with additional significant constraint:

Now i have 2 tables, one for measurements and one for classified results for part of the measurements.

measurements are constantly arrive so as result, that are constantly added after classification of new measurements.

results will not necessarily be stored in the same order of measurement's arrive and store order!

I am interested only to present the last results. By last i mean to take the max time (the time is a part of the measurement structure) of last available result call it Y and a range of X seconds , and present the measurements together with the available results in the range beteen Y and YX.

The following are the structure of 2 tables:

event table:

CREATE TABLE `event_data` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `Feature` char(256) NOT NULL,
  `UnixTimeStamp` int(10) unsigned NOT NULL,
  `Value` double NOT NULL,

  KEY `ix_filter` (`Feature`),
  KEY `ix_time` (`UnixTimeStamp`),
  KEY `id_index` (`id`)
) ENGINE=MyISAM

classified results table:

CREATE TABLE `event_results` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `level` enum('NORMAL','SUSPICIOUS') DEFAULT NULL,
  `score` double DEFAULT NULL,
  `eventId` int(11) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `eventId_index` (`eventId`)
) ENGINE=MyISAM

I can't query for the last measurements timestamp first since i want to present measurements for which there are currently results, and since measurements arrive constantly, results may still not be available.

Therefore i thought of joining the two tables using
event_results.eventId=event_data.id and than selecting the max time of event_data.UnixTimeStamp as maxTime , after i have the maxTime, i need to do the same opearation again (joining 2 tables) and adding in a where clause a condition

WHERE event_data.UnixTimeStamp >= maxTime + INTERVAL -X SECOND

It seems to be not efficient to execute 2 joins only to achieve what i am asking, Do you have more ef

Answer 1

From my understanding, you are using an aggregate function, MAX . This will produce a record set of size one as a result, which is the highest time from which you will perform. Therefore, it needs to be broken out into a sub query (As you say, nested select). You HAVE to do 2 queries at some point. (Your answer to the last question has 2 queries in it, by having subqueries/nested selects).

The main time sub queries cause problems is when you perform the subquery in the select part of the query, as it performs the subquery for each time there is a row, which will make the query run exponentially slower as the resultset grows. Lets take the answer to your last question and write it in a horrible, inefficient way:

SELECT timeStart, 
       (SELECT max(timeStart) FROM events) AS maxTime
FROM events
WHERE   timeStart > (maxTime + INTERVAL -1 SECOND)

This will perform a select query for each time there is an eventTime record, for the max eventtime. It should produce the same result, but this is slow. This is where the fear of subqueries comes from.

It also performs the aggregate function MAX on each row, which will return the same answer each time. So, you perform that sub query ONCE rather than on each row.

However, in the case of the answer of your last question, the MAX sub query part is ran once, and used to filter on the where, of which that select is ran once. So, in total, 2 queries are ran.

2 super fast queries are faster ran one after the other than 1 super slow query that is super slow.

Answer 2

I'm not entirely sure what resultset you want returned, so I am going to make some assumptions. Please feel free to correct any assumptions I've made.

It sounds (to me) like you want ALL rows from event_data that are within an hour (or however many seconds) of the absolute "latest" timestamp, and along with those rows, you also want to return any related rows from event_results , if any matching rows are available.

If that's the case, then using an inline view to retrieve the maximum value of timestamp is the way to go. (That operation will be very efficient, since the query will be returning a single row, and it can be efficiently retrieved from an existing index.)

Since you want all rows from a specified period of time (from the "latest time" back to "latest time minus X seconds"), we can go ahead and calculate the starting timestamp of the period in that same query. Here we assume you want to "go back" one hour (=60*60 seconds):

SELECT MAX(UnixTimeStamp) - 3600 FROM event_data

NOTE: the expression in the SELECT list above is based on UnixTimeStamp column defined as integer type, rather than as a DATETIME or TIMESTAMP datatype. If the column were defined as DATETIME or TIMESTAMP datatype, we would likely express that with something like this:

SELECT MAX(mydatetime) + INTERVAL -3600 SECONDS

(We could specify the interval units in minutes, hours, etc.)

We can use the result from that query in another query. To do that in the same query text, we simply wrap that query in parentheses, and reference it as a rowsource, as if that query were an actual table. This allows us to get all the rows from event_data that are within in the specified time period, like this:

SELECT d.id
     , d.Feature
     , d.UnixTimeStamp
     , d.Value
  JOIN ( SELECT MAX(l.UnixTimeStamp) - 3600 AS from_unixtimestamp
           FROM event_data l
       ) m
  JOIN event_data d
    ON d.UnixTimetamp >= m.from_unixtimestamp

In this particular case, there's no need for an upper bound predicate on UnixTimeStamp column in the outer query. This is because we already know there are no values of UnixTimeStamp that are greater than the MAX(UnixTimeStamp), which is the upper bound of the period we are interested in.

(We could add an expression to the SELECT list of the inline view, to return MAX(l.UnixTimeStamp) AS to_unixtimestamp , and then include a predicate like AND d.UnixTimeStamp <= m.to_unixtimestamp in the outer query, but that would be unnecessarily redundant.)

You also specified a requirement to return information from the event_results table.

I believe you said that you wanted any related rows that are "available". This suggests (to me) that if no matching row is "available" from event_results , you still want to return the row from the event_data table.

We can use a LEFT JOIN operation to get that to happen:

SELECT d.id
     , d.Feature
     , d.UnixTimeStamp
     , d.Value
     , r.id
     , r.level
     , r.score
     , r.eventId
  JOIN ( SELECT MAX(l.UnixTimeStamp) - 3600 AS from_unixtimestamp
           FROM event_data l
       ) m
  JOIN event_data d
    ON d.UnixTimetamp >= m.from_unixtimestamp
  LEFT
  JOIN event_results r
    ON r.eventId = d.id

Since there is no unique constraint on the eventID column in the event_results table, there is a possibility that more than one "matching" row from event_results will be found. Whenever that happens, the row from event_data table will be repeated, once for each matching row from event_results .

If there is no matching row from event_results , then the row from event_data will still be returned, but with the columns from the event_results table set to NULL.

For performance, remove any columns from the SELECT list that you don't need returned, and be judicious in your choice of expressions in an ORDER BY clause. (The addition of a covering index may improve performance.)

For the statement as written above, MySQL is likely to use the ix_time index on the event_data table, and the eventId_index index on the event_results table.