将日期时间字段添加到RecArray

Question

I am trying to append a date-time field (datetime64) to an existing recarray - without much success. I can create the datetime field, but when I attempt to append it to the record array I get the error:

ValueError: Error parsing datetime string "?" at position 0

However, if I cast the data as int64 I can add it in that format without problem. (code shown below)

Anyone know why this does not work?

(my ultimate goal is to write the recarray to a netcdf file, so an appropriate datetime format with that goal in mind would also be helpful)

I am using python 2.7.6.1, numpy 1.8.1

Thanks, Rob

import numpy as np
import numpy.lib.recfunctions as rf

# ----- make a recarray ---------    
dummy = np.arange(0,10)
datarray = np.core.records.fromarrays([dummy,dummy,dummy],names='a,b,c')

# ----- make some time data using datetime64 ---------
sec = np.arange(0,10)*1000
millisec = np.arange(0,10) 
mytime = sec + millisec

mytime64 = mytime.astype('timedelta64[ms]')         
basetime =  np.datetime64('1990-01-01') 
mydatetime = mytime64+basetime  

# ----- convert time data to int64 ---------
idatetime = mydatetime.astype('int64');

#------ try and append to recarray ---------
#  this works
datarray = rf.append_fields(datarray, 'iDateTime', data=idatetime)
# this doesnt
datarray = rf.append_fields(datarray, 'DateTime', data=mydatetime)

Answer 1

The traceback is:

Traceback (most recent call last):
  File "stack26739733.py", line 30, in <module>
    datarray = rf.append_fields(datarray, 'DateTime', data=mydatetime, usemask=False, dtypes=mydatetime.dtype)
  File "/usr/local/lib/python2.7/site-packages/numpy/lib/recfunctions.py", line 641, in append_fields
    dtype=base.dtype.descr + data.dtype.descr)
  File "/usr/local/lib/python2.7/site-packages/numpy/ma/extras.py", line 163, in masked_all
    mask=np.ones(shape, make_mask_descr(dtype)))
  File "/usr/local/lib/python2.7/site-packages/numpy/ma/core.py", line 2644, in __new__
    _data = ndarray.view(_data, cls)
  File "/usr/local/lib/python2.7/site-packages/numpy/ma/core.py", line 2800, in __array_finalize__
    self._fill_value = _check_fill_value(None, self.dtype)
  File "/usr/local/lib/python2.7/site-packages/numpy/ma/core.py", line 402, in _check_fill_value
    dtype=ndtype,)
ValueError: Error parsing datetime string "?" at position 0

So this append function constructs a masked array ( ma ), and checks the 'fill_value' for the appended 'dtype'. Apparently _check_fill_value doesn't understand the datetime dtype. Looks like an incompatibility between masked array and datetime. A numpy bug report might be in order.

---

Here's a simple, do-it-yourself append:

dt1 = np.dtype(datarray.dtype.descr + mydatetime.dtype.descr)
newarray = np.empty(datarray.shape, dtype=dt1)
for n in datarray.dtype.names:
    newarray[n] = datarray[n]
newarray['f3'] = mydatetime

I construct an empty array with a union dtype. Then I copy the data from both datarray and mydatetime field by field. Since the number of fields is normally quite small compared to the shape , this copy is quite fast. I'm pretty sure the rf function does the same.

'f3' is the default name of the added field. You can change that when creating dt1 .

The result is:

array([(0, 0, 0, datetime.datetime(1990, 1, 1, 0, 0)),
       (1, 1, 1, datetime.datetime(1990, 1, 1, 0, 0, 1, 1000)),
       (2, 2, 2, datetime.datetime(1990, 1, 1, 0, 0, 2, 2000)),
       ...
       (9, 9, 9, datetime.datetime(1990, 1, 1, 0, 0, 9, 9000))], 
      dtype=[('a', '<i4'), ('b', '<i4'), ('c', '<i4'), ('f3', '<M8[ms]')])

Turning this newarray into a masked array produces the same _check_fill_value error.

np.ma.masked_array(newarray)