How to optimize database timezones with convert_tz or any other function

Question

I am trying to optimise my mySQL query when handling timezones. My database (mySQL) is set to EET time(+02:00) (I will soon move on AWS where I will use UTC), but in any case, our Cakephp implementation has a setting that retrieves the records as UTC. Our timestamp column his a timestamp type.

So a 2019-12-19 12:44:27 found in our mySQL (+2), is actually 2019-12-19 10:44:27 (UTC) within our CakePHP implementation.

The thing is that I need to display rows between date ranges, for example today's results BUT according to the company's timezone and not according to server/database.

I have created the following query considering a +04:00 timezone.

$company_timezone ='+04:00';
SELECT company_id, COUNT( timestamp ) AS views, url 
FROM behaviour 
WHERE company_id = 1
AND CONVERT_TZ(timestamp,'+00:00','{$company_timezone}')  >= DATE(CONVERT_TZ(NOW(),'+00:00','{$company_timezone}')) 
GROUP BY URL 
ORDER BY views 
DESC LIMIT 20

However this is quite needy in terms of performance. It takes approx 4-5 seconds. Without the convert_tz it takes no more than 0.5 sec.

My question is how can I optimise this? Of course, our timestamp column is indexed, even it doesn't make any sense at the specific query because I use it with convert_tz .

new query that deals with date range (today - yesterday)

I believe the following example emphasizes more on my case: With this query, I show the results based on the company's timezone. For example, if their company's local time is 00:01 then it is a new day for them, regardless the time of server/mysql.

SELECT COUNT(hash) as how_many
FROM   table          
WHERE  company_id = 1
AND CONVERT_TZ(last_visit,'+00:00','{$company_timezone}')  >= DATE(CONVERT_TZ(NOW(),'+00:00','{$company_timezone}') - INTERVAL 1 DAY)
GROUP BY date(last_visit)
ORDER BY last_visit DESC

Thank you

Answer 1

From the MySQL documentation about the TIMESTAMP type :

MySQL converts TIMESTAMP values from the current time zone to UTC for storage, and back from UTC to the current time zone for retrieval. (This does not occur for other types such as DATETIME .) By default, the current time zone for each connection is the server's time. The time zone can be set on a per-connection basis.

Thus you are actually saving UTC-based values. You may think you've stored UTC+2 values, but that's just because the default time zone at query time (the session time zone) is the same as the server's default time zone.

Additionally, functions like NOW() also use the session time zone. Thus, since the session time zone is the same on both sides, there is no time zone conversion to be done. You can just say:

AND timestamp >= NOW()

This has the added benefit (as symcbean's answer pointed out), of allowing the DBMS to use an index - in other words, the query becomes sargable .

You might also want to read the excellent article in the MySql documentation, Indexed Lookups from TIMESTAMP Columns , which explains how the session time zone impacts both indexed and non-indexed queries.

It might also be worthwhile to set the time zone explicitly to UTC at time of query:

SET time_zone = 'UTC';

You'll get the same results either way, but this is slightly more efficient as the DBMS now has fewer time zone conversions to perform.

---

For the second query in your edit, you still are showing a conversion on the left-side of the expression. Again, you want that to be just the field so that an index can be used. Since the field is a TIMESTAMP type, you simply need to calculate the earliest timestamp you want to retrieve.

If you want to do this inline your query, the number of conversions make it a bit convoluted:

SELECT COUNT(hash) as how_many
FROM   table          
WHERE  company_id = 1
  AND  last_visit >= CONVERT_TZ(DATE(CONVERT_TZ(NOW(), @@session.time_zone, '{$company_timezone}')) - INTERVAL 1 DAY, '{$company_timezone}', @@session.time_zone)
GROUP BY DATE(CONVERT_TZ(last_visit, @@session.time_zone, '{$company_timezone}'))
ORDER BY DATE(CONVERT_TZ(last_visit, @@session.time_zone, '{$company_timezone}'))

For this reason, it's a lot easier to simply change the session time zone before your query such that it applies uniformly to all operations:

SET time_zone = '{$company_timezone}';
SELECT COUNT(hash) as how_many
FROM   table          
WHERE  company_id = 1
  AND  last_visit >= DATE(NOW()) - INTERVAL 1 DAY
GROUP BY DATE(last_visit)
ORDER BY DATE(last_visit)

Answer 2

I have two solution for your question, hope one of them works for you

Solution 1

Change timeZone of datetime in cakePHP and then execute query.

public static function convertDate($datetime, $companyTimeZone, $dbTimeZone)
{
    $newDate = new DateTime($datetime, new DateTimeZone($companyTimeZone));

    $newDate->setTimezone(new DateTimeZone($dbTimeZone));
    return $newDate->format('Y-m-d H:i:s');
}
$datetime = self::convertDate($datetime, $companyTimeZone, $dbTimeZone);

SELECT company_id, COUNT( timestamp ) AS views, url 
FROM behaviour 
WHERE company_id = 1
AND timestamp  >= $datetime 
GROUP BY URL 
ORDER BY views 
DESC LIMIT 20

Solution 2

You can set timeZone of mysql for the current session only

##$company_timezone ='+04:00';

#NOTE this will set timZone only for the current session
SET time_zone= $company_timezone;

SELECT company_id, COUNT( timestamp ) AS views, url 
FROM behaviour 
WHERE company_id = 1
AND timestamp  >= NOW() 
GROUP BY URL 
ORDER BY views 
DESC LIMIT 20

Note : I have my application whose DB is UTC and users are coming from all over the globe, So I have implemented Solution 1 and it is working great since 2 year.

Answer 3

You are applying the same transformation to both sides of the predicate - all this does is make it impossible for the DBMS to use the index. Compare the values directly:

AND timestamp >= NOW()

This gets a bit more complicated if you are comparing data in one timezone with a wallclock in a different timezone - but you just need to make sure that you only apply the transformation to the side of the predicate which only contains literals, and not attribute references.

Answer 4

我建议将时间戳转换为（纪元转换）长值并将其保存到临时列，然后将长值转换为时区特定的时间戳。

Answer 5

I would suggest you a real DIRTY method !

This method, will don't alter your database, and will avoid any big computation (will not use the convert_tz ) on the DB. But it will require some computation on the back-office (or front-office if you wan't to be more dirty...).

The dirty trick here is that you will do your time conversion only on the result and not on all data which are present in your db.

---

First you have to compute (and keep) the maximum time interval between :

• the company timezone
• the cakePHP server timezone
• DB server timezone

and

• the UTC timezone.

In your case it may be: 4H .

Then

1- Add 4H to your target end_date , then convert this date to a end_timestamp , keep your start_date - without any modification - and convert this date to a start_timestamp .

2- Make your query using start_timestamp & end_timestamp . [This avoid any timezone and time conversion in your query]

Note that you will get too much data (you will get "junk" data more especially near the start_timestamp ) in your DB query result ...

2- On your back-office : Filter bad element ( which mean you will have to remove all data which are between the previous start_timestamp and this same start_timestamp +4H ) to remove all junk data.

3- On your back-office : Finally convert the remaining results to the good timezone (UTC+4 in your case)

4- Send the data to the front-office .

Answer 6

thanks to the fact that the column is already of the type timezone , you don't need to use convert_tz . MySQL will always work with the current set time zone (that's one of the differences to the column type datetime ).

So, you can set the timezone beforehand and then query without convert_tz . All indexes should be used.

SET timezone = '{$company_timezone}';

SELECT COUNT(hash) as how_many
FROM   table          
WHERE  company_id = 1
AND last_visit >= DATE(NOW() - INTERVAL 1 DAY)
GROUP BY DATE(last_visit);
ORDER BY last_visit DESC

You may use correct time zones like 'US/Eastern' directly. With this you will handle summer and winter time without any hassle. Here is how to set it up: https://dev.mysql.com/doc/refman/8.0/en/time-zone-support.html