Python Timestamp + Datetime Conversion

Question

I have a test record with a UNIX millisecond timestamp since epoch value of 1459952824358

I am writing Python 2.x program using the fastavro library. That takes the 1459952824358 integer UNIX millisecond timestamp and converts it to a local datetime value of 2016-04-06T09:27:04.358000 . When I try to convert that back to a integer UNIX millisecond timestamp value I end up one hour off. I can replicate this conversion problem outside of the fastavro library with plain Python datetime:

import datetime
import time

raw = 1459952824358

local_dt = datetime.datetime.fromtimestamp(raw / 1000.0)
local_dt.isoformat() # '2016-04-06T09:27:04.358000'

back = int((local_dt - datetime.datetime.fromtimestamp(0)).total_seconds() * 1.0e3)
back - raw # 3600000

I can convert to a UTC datetime and back successfully:

import datetime
import time

raw = 1459952824358

utc_dt = datetime.datetime.utcfromtimestamp(raw / 1000.0)
utc_dt.isoformat() # '2016-04-06T14:27:04.358000'

back = int((utc_dt - datetime.datetime.utcfromtimestamp(0)).total_seconds() * 1.0e3)
back - raw # 0

What is odd is that my system is set to CST or UTC-6, yet Python produces a local datetime that is five hours off of the UTC value rather than six.

While, my specific project requires Python 2 compatibility, I try the Python 3 timestamp function which correctly gives me back the original timestamp value without the hour drift, but the local datetime is still bizarrely 5 hours off of the UTC time, even when my system is set to CST or UTC-6.

import datetime
import time

raw = 1459952824358

local_dt = datetime.datetime.fromtimestamp(raw / 1000.0)
local_dt.isoformat() # '2016-04-06T09:27:04.358000'

back = int(local_dt.timestamp() * 1000)
back - raw # 0

I try the same example in Java 9 on my same laptop, just to see what it produces:

import java.time.*

long raw = 1459952824358L;

Instant i = Instant.ofEpochMilli(raw)
ZonedDateTime zdtUtc = ZonedDateTime.ofInstant(i, ZoneOffset.UTC)
// UPDATE: This line was my original attempt. It was wrong because it uses the current daylight savings setting, not the one of the given date.
ZonedDateTime zdtLocal = zdtUtc.withZoneSameInstant(ZonedDateTime.now().getOffset())
// This next line will choose the current geography, but choose the daylight savings setting based on the date, which is what I expected.
ZonedDateTime zdtLocal = zdtUtc.withZoneSameInstant(ZoneId.systemDefault())

long deltaFromUtc = zdtUtc.toInstant().toEpochMilli() - raw
long deltaFromLocal = zdtLocal.toInstant().toEpochMilli() - raw

It returns back the exact original timestamp value regardless of utc/local.

UPDATE: Basically, Python 2.x can't properly convert a local datetime value to UTC without extra libraries. Python 3.x can, and Python 2.x has extra libraries to do this, but out of the box, it can't.

Answer 1

tl;dr : Your measurement is off, not the datetimes: You're comparing two times from different timezones, so the result should be the difference between the timezones, and it is.

---

So this is why it's better to work in UTC:

In Python 2.7:

Using your raw:

>>> print datetime.datetime.utcfromtimestamp(raw/1000.0)
... print datetime.datetime.fromtimestamp(raw/1000.0)
... 
2016-04-06 14:27:04.358000
2016-04-06 07:27:04.358000

7 hrs different

Using a modified date outside Daylight Savings Time:

>>> print datetime.datetime.utcfromtimestamp(1456952824)
... print datetime.datetime.fromtimestamp(1456952824)
... 
2016-03-02 21:07:04
2016-03-02 13:07:04

8 hours different

This is happening because raw and 0 in the local time are in different timezones (Daylight Savings Time vs Standard):

>>> print datetime.datetime.utcfromtimestamp(raw/1000.0)
... print datetime.datetime.fromtimestamp(raw/1000.0)
... 
2016-04-06 14:27:04.358000
2016-04-06 07:27:04.358000

>>> print datetime.datetime.utcfromtimestamp(0)
... print datetime.datetime.fromtimestamp(0)
...
1970-01-01 00:00:00
1969-12-31 16:00:00

If you stick to utc, no issues.

but the local datetime is still bizarrely 5 hours off of the UTC time, even when my system is set to CST or UTC-6.

Because on that raw date, your local time would be CDT (-5):

user@TS-E31:~$ date --date @1459952824
Wed Apr  6 07:27:04 PDT 2016
user@TS-E31:~$ date --date @0
Wed Dec 31 16:00:00 PST 1969