Handling CSV with timezone-aware and timezone-naive datetime column

Question

I have a pandas dataframe that is imported from a csv that looks like this:

|date time|id|value|
|------|-------|---------|
|2019-10-08T01:00:00+01:00|1|35|
|2019-10-08T02:00:00+01:00|1|32|
|2019-10-08T03:00:00+01:00|1|33|
|2019-12-08T01:00:00Z|1|25|
|2019-12-08T01:00:00Z|1|15|
|2019-12-08T01:00:00Z|1|25|

When I try to do an aggregation like this:

data.groupby([data['Date'].dt.date]).agg(['mean', 'count'])

I get an error like this:

ValueError: Cannot mix tz-aware with tz-naive values

An additional wrinkle is that, it is important to use these date values and not the UTC values as I would be doing peak-hour analysis based on the local (British) time. What's the best way to fix this?

Answer 1

for given example with column date time as string datatype,

df['date time']
0    2019-10-08T01:00:00+01:00
1    2019-10-08T02:00:00+01:00
2    2019-10-08T03:00:00+01:00
3         2019-12-08T01:00:00Z
4         2019-12-08T01:00:00Z
5         2019-12-08T01:00:00Z
Name: date time, dtype: object

convert to datetime datatype using pd.to_datetime with keyword utc=True , then convert to the appropriate time zone:

df['date time'] = pd.to_datetime(df['date time'], utc=True).dt.tz_convert('Europe/London')

to get

df['date time']
0   2019-10-08 01:00:00+01:00
1   2019-10-08 02:00:00+01:00
2   2019-10-08 03:00:00+01:00
3   2019-12-08 01:00:00+00:00
4   2019-12-08 01:00:00+00:00
5   2019-12-08 01:00:00+00:00
Name: date time, dtype: datetime64[ns, Europe/London]

Now the groupby works as intended:

df.groupby([df['date time'].dt.date]).agg(['mean', 'count'])
             id            value      
           mean count       mean count
date time                             
2019-10-08    1     3  33.333333     3
2019-12-08    1     3  21.666667     3