Create an indexed datetime from date/time info in 3 columns using pandas

Question

First off, here is a sample of my data, a csv with Year, Julian Day, 2400hr, and then 2 value columns.

2014,92,1931,6.234,10.14
2014,92,1932,5.823,9.49
2014,92,1933,5.33,7.65
2014,92,1934,4.751,6.19
2014,92,1935,4.156,5.285
2014,92,1936,3.962,4.652
2014,92,1937,3.74,4.314
2014,92,1938,3.325,3.98
2014,92,1939,2.909,3.847
2014,92,1940,2.878,3.164

So, I start by loading libraries

    import numpy as np
    import matplotlib.pyplot as plt
    import pandas as pd
    from datetime import datetime

then I run the parser (correct me if i'm wrong, this is the bit I format to match my data?)

def parser(x):
    return pd.datetime.strptime(x, '%Y %j %H%M')

then I go to create the variable "data", which is a read in and, hopefully, datetime indexed pandas dataframe...

data = pd.read_csv('sorted.dat',parse_dates=[0,1,2], date_parser=parser,index_col=0, header=None )

The resulting data frame looks like this:

dt  3   4
0   2014 92 1931    6.234   10.140
1   2014 92 1932    5.823   9.490
2   2014 92 1933    5.330   7.650
3   2014 92 1934    4.751   6.190
4   2014 92 1935    4.156   5.285
5   2014 92 1936    3.962   4.652
6   2014 92 1937    3.740   4.314
7   2014 92 1938    3.325   3.980
8   2014 92 1939    2.909   3.847
9   2014 92 1940    2.878   3.164
10  2014 92 1941    2.303   3.020
11  2014 92 1942    2.078   2.700
12  2014 92 1943    2.078   2.161
13  2014 92 1944    1.784   2.157
14  2014 92 1945    1.319   1.902
15  2014 92 1949    1.077   1.294
16  2014 92 1950    0.838   1.262
17  2014 92 1951    0.703   0.949
18  2014 92 1952    0.436   0.834
19  2014 92 1953    0.416   0.564
20  2014 92 1954    0.416   0.431
21  2014 92 1955    0.416   0.431
22  2014 92 1956    0.416   0.431
23  2014 92 1957    0.416   0.431
24  2014 92 1958    0.416   0.431
25  2014 92 1959    0.416   0.431
26  2014 92 2000    0.416   0.431
27  2014 92 2001    0.416   0.431
28  2014 92 2002    0.405   0.431
29  2014 92 2003    0.360   0.421
... ... ... ...
337887  2014 355 2330   0.000   0.000
337888  2014 355 2331   0.000   0.000
337889  2014 355 2332   0.000   0.000
337890  2014 355 2333   0.000   0.000
337891  2014 355 2334   0.000   0.000
337892  2014 355 2335   0.000   0.000
337893  2014 355 2336   0.000   0.000
337894  2014 355 2337   0.000   0.000
337895  2014 355 2338   0.000   0.000
337896  2014 355 2339   0.000   0.000
337897  2014 355 2340   0.000   0.000
337898  2014 355 2341   0.000   0.000
337899  2014 355 2342   0.000   0.000
337900  2014 355 2343   0.000   0.000
337901  2014 355 2344   0.000   0.000
337902  2014 355 2345   0.000   0.000
337903  2014 355 2346   0.000   0.000
337904  2014 355 2347   0.000   0.000
337905  2014 355 2348   0.000   0.000
337906  2014 355 2349   0.000   0.000
337907  2014 355 2350   0.000   0.000
337908  2014 355 2351   0.000   0.000
337909  2014 355 2352   0.000   0.000
337910  2014 355 2353   0.000   0.000
337911  2014 355 2354   0.000   0.000
337912  2014 355 2355   0.000   0.000
337913  2014 355 2356   0.000   0.000
337914  2014 355 2357   0.000   0.000
337915  2014 355 2358   0.000   0.000
337916  2014 355 2359   0.000   0.000

When I run this I get an error

  ValueError: time data 'dt' does not match format '%Y %j %H%M'

Answer 1

Try adding a parser to your read_csv

#assuming the order is year, month, day.  if you have time too, '%Y-%m-%d %H:%M:%S'    
parser = lambda p: pd.datetime.strptime(p, '%Y-%m-%d')  

df = pd.read_csv('sorted.dat', 
                  parse_dates={'datetime': [1,2,3]}, 
                  date_parser=parser, 
                  header=None)

Update

Parser looks correct. I believe your current problem is in your read_csv(). The parse_dates arg is not formatted corrected (see excerpt from the doc string below).

If [1, 2, 3] -> try parsing columns 1, 2, 3 each as a separate date column.

If [[1, 3]] -> combine columns 1 and 3 and parse as a single date column.

So your parser was expecting all 3 columns at once, but was getting them 1 at a time. I discovered this when I added a print x statement to the parser func. Try this modification that uses a list of lists approach

data = pd.read_csv('sorted.dat',parse_dates=[[0,1,2]], date_parser=parser,index_col=0, header=None )