How to index an epoch nanoseconds value as a date or date_nano in Elasticsearch?

Question

Our system stores its values in Epoch nanoseconds timestamps, ie "created_at": 1629469976984334600 , which are imported as such in an Elasticsearch index.

When the field is mapped on a new index, I get unexpected results:

• mapped as "type": "date_nanos" → the original integer numbers (?)

• mapped as "type": "date" → dates in year 51,213,565-ish, eg +51213565-09-01T05:06:38.320Z

Is there some mapping syntax that tells ES to treat the nanoseconds number as a proper date, even if the result is milliseconds or seconds precision?

Answer 1

Did you try to simply add the nanoseconds precision to the mapping itself, something like this -

PUT new_index
{
  "mappings": {
    "properties": {
      "new_date_field": {
        "type": "date",
        "format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ"
      }
    }
  }
}

Please refer to elastic's built-in date formats

Answer 2

After too much digging, a failed bounty and more digging, I found a way.

In a _reindex , you can use a script to manipulate the data, like this:

POST _reindex
{
  "source": {
    "index": "mydocs"
  },
  "dest": {
    "index": "mydocs_mapped"
  },
  "script": {
    "lang": "painless",
    "inline": "ctx._source.created_at = (long)ctx._source.created_at / 1000000;"
  }
}

This converts the field to epoch milliseconds and Elasticsearch and Kibana can use it as a date.

I almost teared up.

PS: Official documentation is here https://www.elastic.co/guide/en/elasticsearch/reference/7.10/docs-reindex.html#reindex-scripts