Mysql返回最后一个活动早于X天的ID

Question

i have a log tabel where every user activity is stored.

UserActivityTable (arround 15Milion records)

  id    userID    category           value                 timestamp 
    1        2         Visit          homepage          2018-02-21 13:13:54
    1        2         Visit          page2             2018-02-18 13:13:45
    1        2         Visit          page1             2018-02-15 13:13:30
    1        3         Visit          homepage          2018-02-01 13:13:12

With an SQL query i need to get all userID´s where the last activity is older than X Days (lets say 30) if the user is set to "Active"

Users(arround 15k User)

id     Groups     Active   Name    Mails ...
2      Customer    1       Hans
3      Customer    0       Wurst

if i get all user that active (arround 5k) and than try to get there last activity i run into a timeout (the query is not perfomant i think) If i limit it to 5 there is no problem.

What i try.

1 Select of all User that are active, than use a foreach function to get there last activity if its older than 30 days i write it inside new array and at the end i use that array to set the activity inside the user table to false.

Untill the last 2-3 Month it was just fine but now we have a lot of new users and the function cant handle it.

Is there a clean way to get all that stuff in one sql query?

Answer 1

You can use the following query to get the Users :

SELECT `userID`, MAX(`timestamp`) AS lastActive FROM `UserActivityTable` 
WHERE `userID` IN (
    SELECT `id` FROM `Users` WHERE `Active` = 1
) GROUP BY `userID` HAVING lastActive < DATE_SUB(NOW(), INTERVAL 30 DAY)

Indexing

• You should use a PRIMARY KEY index on the Users table.
• You should use a FOREIGN KEY index on UserActivityTable table.
• To speed up the query above you can create a column index on timestamp column.

You can use the following to create a INDEX on the timestamp column:

CREATE INDEX index_timestamp ON `UserActivityTable` (`timestamp`);

---

You can also use a single query to UPDATE the active state on the Users table:

UPDATE `Users` SET `active` = EXISTS (
    SELECT `userID` FROM `UserActivityTable` WHERE `UserActivityTable`.`userID` = `Users`.`id` GROUP BY `UserActivityTable`.`userID` HAVING MAX(`UserActivityTable`.`timestamp`) > DATE_SUB(NOW(), INTERVAL 30 DAY)
)

Answer 2

Is there a clean way to get all that stuff in one sql query?

Yes , you can update the Users table in a single step with the following query:

UPDATE `Users` SET `Active` = EXISTS(
    SELECT * from `UserActivityTable ` WHERE
        `UserActivityTable `.`userID` = `Users`.`id` AND
        `timestamp`>DATE_SUB( NOW(), INTERVAL 30 DAY )
    )

The EXISTS statement returns 1 or 0 depending if does esists at least one record in the user activity in the last 30 days. So the field Active is properly updated to 1 or 0 for every user.

---

Mysql Return ID where last activity is older than X Days

If you just want the list of users' id with activity you have:

SELECT `Users`.`id` WHERE EXISTS(
    SELECT * from `UserActivityTable ` WHERE
        `UserActivityTable `.`userID` = `Users`.`id` AND
        `timestamp`>DATE_SUB( NOW(), INTERVAL 30 DAY )
    ) = 1

---

In order to have good performaces (at least) the field timestamp must be indexed.

---

Side note

You already hit 15M records.

As your events table will grow indefinitely over time you should consider deleting periodically old entries or moving them to a separate table/dump file.

Answer 3

Don't do it.

It is improper to have redundant information in a database. ( active is redundant because it can be discovered by a query against UserActivityTable .)

OK, you need more performance, so you are setting this flag. I assume this is not a one-time task, but needs to be updated daily? Or what? I ask this because active=0 will be wrong if the 'user' does something after you run the UPDATE , and before running it again!

Let's solve that bug , then discover that we are making the UPDATE very fast in the process.

The 'only' way to fix that bug is to reach into UserActivityTable dynamically. However, we can make that so cheap that it is OK to do it in 'realtime'.

FROM Users
WHERE EXISTS ( SELECT * FROM UserActivityTable
                 WHERE userID = x.userID
                   AND timestamp > NOW() - INTERVAL 30 DAY )  -- == "active"

UserActivityTable needs INDEX(userID, timestamp)

Oops! I just obviated the need for the active column.

One of your Comments mentioned purging 'old, inactive' users?? Is the UPDATE aimed at that? Please fold that requirement into the question, else I (and others) are not necessarily helping you.