python pandas to_datetime is not working steadily

Question

I usually call data from some csv file and use pandas.to_datetime function to change the date columns to datetime format for further data processing.

However, sometimes the to_datetime funcion works well and sometimes not. It is not working steadily and I usually use a lot of time to adjust the datatime format..

I tried a lot of methods but they all dont work steadily. Would someone please kindly help me to solve this problem?

df1 = pd.read_csv("somefile.csv", encoding='utf-8', parse_dates=[0])
# the result turns out that the parse_dates does not work at all here

df1["Date"]= df1["Date"].apply(pd.to_datetime)
# after this change, the type(df1["Date"][0]) becomes pandas._libs.tslib.Timestamp

df1["Date"] = df1["Date"].dt.date.apply(lambda x: datetime.date(x.year,x.month,x.day))
# this code worked yesterday but not today anymore...
# TypeError: descriptor 'date' requires a 'datetime.datetime' object but received a 'int'

The error code here says "descriptor 'date' requires a 'datetime.datetime' object but received a 'int'" I would like to change the type of df["Date"] to datetime type instead of timestamp.

My pandas dataframe looks like this(only the date column displayed) The raw data is provided here: https://www.dropbox.com/s/rrhy9my9yp1gy2y/test.csv?dl=0

   Date
2015-01-07
2015-01-08
2015-01-09
2015-01-10
2015-01-11

My python version is 2.7 Got frustrated with this problem for awhile, People except me everyones to_datetime function works well?

Answer 1

You need remove apply , need only Series.dt.date :

df1 = pd.read_csv('test.csv', parse_dates=[0])

df1["Date"] = df1["Date"].dt.date.apply(lambda x: datetime.date(x.year,x.month,x.day))

to:

df1["Date"] = df1["Date"].dt.date

print (type(df1.loc[0, 'Date']))
<class 'datetime.date'>

But if need tuples:

df1["Date"] = df1["Date"].dt.date.apply(lambda x: (x.year,x.month,x.day))
print (df1.head())
            Date  trend
0   (2015, 1, 7)     37
1   (2015, 1, 8)     37
2   (2015, 1, 9)     37
3  (2015, 1, 10)     37
4  (2015, 1, 11)     38