Datetime error in combining date and time dataframe objects

Question

I have an issue with Python when I want to merge two data columns into one DateTime object. The initial dates column is in string format and hours are integers (1, 2, 3, ....,23, 24) and a new day starts with 1 again (not with 24). I use the command smartmeter_data['Datetime']=pd.to_datetime(smartmeter_data['Date']) + smartmeter_data['Time'].astype('timedelta64[h]') to add a new column with both: date and time. However, I received very weird results:

...
19  01/09/2019 2019-01-09 20:00:00
20  01/09/2019 2019-01-09 21:00:00
21  01/09/2019 2019-01-09 22:00:00
22  01/09/2019 2019-01-09 23:00:00
23  01/09/2019 2019-01-10 00:00:00
24  02/09/2019 2019-02-09 01:00:00
25  02/09/2019 2019-02-09 02:00:00
26  02/09/2019 2019-02-09 03:00:00
...

There the date 01/09/2019 was changed to the DateTime object 2019-01-10 00:00:00 , which is wrong and makes a very strange "jump" on my graph. My desired output is:

...
19  01/09/2019 2019-01-09 20:00:00
20  01/09/2019 2019-01-09 21:00:00
21  01/09/2019 2019-01-09 22:00:00
22  01/09/2019 2019-01-09 23:00:00
23  01/09/2019 2019-02-09 00:00:00
24  02/09/2019 2019-02-09 01:00:00
25  02/09/2019 2019-02-09 02:00:00
26  02/09/2019 2019-02-09 03:00:00
...

I tried to find a solution through google but no success. Does anyone know how to solve the issue?

I would be very thankful when you could help, using dates and times is the base of my work.

Answer 1

A day has 24 hours so if you add a timedelta of 24 hours to a date, the date will change to the next day. However, why don't you just subtract 1 to get the correct timedelta (0-23 instead of 1-24)? Eg

import pandas as pd

smartmeter_data = pd.DataFrame({'Date': ['01/09/2019', '01/09/2019', '01/09/2019',
                                         '01/09/2019', '01/09/2019', '02/09/2019',
                                         '02/09/2019', '02/09/2019'],
                                'Time': [20, 21, 22, 23, 24, 1, 2, 3]})

smartmeter_data['Datetime'] = (pd.to_datetime(smartmeter_data['Date'], format='%d/%m/%Y') +
                               (smartmeter_data['Time'] - 1).astype('timedelta64[h]'))

# smartmeter_data
#          Date  Time            Datetime
# 0  01/09/2019    20 2019-09-01 19:00:00
# 1  01/09/2019    21 2019-09-01 20:00:00
# 2  01/09/2019    22 2019-09-01 21:00:00
# 3  01/09/2019    23 2019-09-01 22:00:00
# 4  01/09/2019    24 2019-09-01 23:00:00
# 5  02/09/2019     1 2019-09-02 00:00:00
# 6  02/09/2019     2 2019-09-02 01:00:00
# 7  02/09/2019     3 2019-09-02 02:00:00