使用 Django ORM 每天计算不同时区的事件类型

Question

We have a table that holds multiple events and when they were added. The default timezone used for storing the events is UTC. Eg:

class Events:
    type = models.CharField(max_length=45, null=False)
    date_added = models.DateTimeField(auto_now_add=True)

Now, we want to get a per day count of different event types between two dates - start_date and end_date. Eg: for start_date = "2020-03-1" and end_date = "2020-03-31", output should be -

[{
    "date" : "2020-03-1",
    "event1" : 200,
    "event2" : 606,
    "event3" : 595
},
{
    "date" : "2020-03-2",
    "event1" : 357,
    "event2" : 71,
    "event3" : 634
},
{
    "date" : "2020-03-3",
    "event1" : 106,
    "event2" : 943,
    "event3" : 315
},
{
    "date" : "2020-03-4",
    "event1" : 187,
    "event2" : 912,
    "event3" : 743
},
.
.
.
.
{
    "date" : "2020-03-31",
    "event1" : 879,
    "event2" : 292,
    "event3" : 438
}]

Since the users are in different timezones (America, Europe, Asia, etc), we want to convert the timezone as per user before counting the events. Counting in UTC will have wrong counts per day in the user's timezone. Eg: an event created on 3 March, 1:30 am in IST will be shown on 2 March, 8 pm in UTC and counted accordingly.

It will get really expensive if we do it using for loop. Therefore, we want to do it at the DB level using Django ORM. If not possible to completely depend on Django ORM, we want to make it as efficient as possible.

The best query that we could come up with was:

 Events.objects.filter(
    pk = user_pk, date__range = (
        (end_date - time_delta).strftime("%Y-%m-%d"), 
        end_date.strftime("%Y-%m-%d")
        )
    ).extra({
        "date_added" : "date(date_added)"
    }).values(
        "date_added", 
        "type"
    ).annotate(
        models.Count("type")
    ) 

Where we are getting results like:

<QuerySet [{'date_added': datetime.date(2020, 3, 6), 'type': 'event1', 'type__count': 30}, 
{'date_added': datetime.date(2020, 3, 6), 'type': 'event2', 'type__count': 189}, 
{'date_added': datetime.date(2020, 3, 6), 'type': 'event3', 'type__count': 1}, 
{'date_added': datetime.date(2020, 3, 6), 'type': 'event4', 'type__count': 3}, 
{'date_added': datetime.date(2020, 3, 9), 'type': 'event2', 'type__count': 57}, 
{'date_added': datetime.date(2020, 3, 9), 'type': 'event1', 'type__count': 23}, 
{'date_added': datetime.date(2020, 3, 9), 'type': 'event4', 'type__count': 1}, 
{'date_added': datetime.date(2020, 3, 10), 'type': 'event1', 'type__count': 5}, 
{'date_added': datetime.date(2020, 3, 10), 'type': 'event2', 'type__count': 21}, 
{'date_added': datetime.date(2020, 3, 11), 'type': 'event2', 'type__count': 9}, 
{'date_added': datetime.date(2020, 3, 11), 'type': 'event1', 'type__count': 15}, 
{'date_added': datetime.date(2020, 3, 12), 'type': 'event2', 'type__count': 49}, 
{'date_added': datetime.date(2020, 3, 13), 'type': 'event2', 'type__count': 8}, 
{'date_added': datetime.date(2020, 3, 13), 'type': 'event1', 'type__count': 3}, 
{'date_added': datetime.date(2020, 3, 17), 'type': 'event1', 'type__count': 16}, 
{'date_added': datetime.date(2020, 3, 17), 'type': 'event2', 'type__count': 26}, 
{'date_added': datetime.date(2020, 3, 17), 'type': 'event4', 'type__count': 1}, 
{'date_added': datetime.date(2020, 3, 17), 'type': 'event3', 'type__count': 1}, 
{'date_added': datetime.date(2020, 3, 18), 'type': 'event2', 'type__count': 64}, 
{'date_added': datetime.date(2020, 3, 18), 'type': 'event1', 'type__count': 11}, 
'...(remaining elements truncated)...']>

This will still require a for loop to get all the events with same date to be added inside one dict, but the timezone issue still persists.

How to solve this?

Answer 1

We were able to solve this at last. We are still using a for loop to get the data in the required format, but we were able to shift the heavy lifting to the DB. Here are a few things first:

1. I am using MySQL DB. (Check the point below to see why this is relevant.)

2. If you want to convert timezone using the time difference directly ("+05:30" for IST or "+00:00" for UTC), you won't need to run any command.

3. However, you will need to run a command to support timezones by name in MySQL DB. Eg: If you are going to use named timezones ("ASIA/KOLKATA" or "UTC"), you will need to run one command:
   • mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -D mysql -u root -p
4. This command is for MySQL DB running in Ubuntu. There are similar
   commands for other DBs like Postgresql and platforms like Windows.

Here is the Django ORM query:

events_list = Events.objects.all().extra(
        {
            "date_added" : "date(CONVERT_TZ(date_added, 'UTC', 'America/Chicago'))"
        }
    ).values(
        "date_added",
        "type"
    ).annotate(
        models.Count(
            "type"
        )
    )

This will give data in the format:

<QuerySet [{'date_added': datetime.date(2020, 3, 6), 'type': 'event1', 'type__count': 31}, 
{'date_added': datetime.date(2020, 3, 6), 'type': 'event2', 'type__count': 189}, 
{'date_added': datetime.date(2020, 3, 6), 'type': 'event3', 'type__count': 1}, 
{'date_added': datetime.date(2020, 3, 6), 'type': 'event4', 'type__count': 3}, 
{'date_added': datetime.date(2020, 3, 9), 'type': 'event2', 'type__count': 58}, 
{'date_added': datetime.date(2020, 3, 9), 'type': 'event1', 'type__count': 21}, 
{'date_added': datetime.date(2020, 3, 9), 'type': 'event4', 'type__count': 1}, 
{'date_added': datetime.date(2020, 3, 10), 'type': 'event1', 'type__count': 1}, 
{'date_added': datetime.date(2020, 3, 10), 'type': 'event2', 'type__count': 23}, 
{'date_added': datetime.date(2020, 3, 11), 'type': 'event2', 'type__count': 10}, 
{'date_added': datetime.date(2020, 3, 11), 'type': 'event1', 'type__count': 16}, 
{'date_added': datetime.date(2020, 3, 12), 'type': 'event2', 'type__count': 50}, 
{'date_added': datetime.date(2020, 3, 13), 'type': 'event2', 'type__count': 10}, 
{'date_added': datetime.date(2020, 3, 13), 'type': 'event1', 'type__count': 1}, 
{'date_added': datetime.date(2020, 3, 17), 'type': 'event1', 'type__count': 19}, 
{'date_added': datetime.date(2020, 3, 17), 'type': 'event2', 'type__count': 27}, 
{'date_added': datetime.date(2020, 3, 17), 'type': 'event4', 'type__count': 3}, 
{'date_added': datetime.date(2020, 3, 17), 'type': 'event3', 'type__count': 1}, 
{'date_added': datetime.date(2020, 3, 18), 'type': 'event2', 'type__count': 61}, 
{'date_added': datetime.date(2020, 3, 18), 'type': 'event1', 'type__count': 13}, 
'...(remaining elements truncated)...']>

Here the number of a particular event is counted after the timezones are converted. Now, once we have the counts for the event after the timezone is converted, all that is remaining is to get this data in the required format which can be easily done with a for loop.

PS:

1. If you have any conditions for the query, you can use filter() instead all. Eg:

    from django.utils import timezone Events.objects.filter( type__in = ["event1", "event2"], date__gt = ( timezone.now() - timezone.timedelta(days = 30) ) ).extra( { "date_added": "date(CONVERT_TZ(date_added, 'UTC', 'ASIA/KOLKATA'))" } ).values( "date_added", "type" ).annotate( models.Count( "type" ) )

   This will give data for last 30 days for event types 1 and 2.

2. You can use "+00:00" for UTC and "+05:30" for ASIA/KOLKATA. Eg: "date_added": "date(CONVERT_TZ(date_added, '+00:00', '+05:30'))"