SQL friendly date format in a JoltTransformJson Processor in Apache Nifi

Question

I need to change the input date to a SQL friendly format in order to insert it into DB. I get errors on both imported_at and processed_at when trying to insert into DB.

My flow: JoltTransformJSON -> ConvertJsonToSql -> PutSql

Input:

{
  "transactionDate": "2018-01-01T18:06:00",
}

My Spec:

[
  {
    "operation": "shift",
    "spec": {
      "transactionDate": "processed_at"
    }
  },
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "processed_at": "=${processed_at.replaceAll('T',' '):toDate('yyyy-MM-dd HH:mm:ss'):format('yyyy-MM-dd HH:mm:ss')}"
    }
  },
  {
    "operation": "default",
    "spec": {
      "processed_at": null,
      "imported_at": "${now():format('yyyy-MM-dd HH:mm:ss')}"
    }
  }
]

My idea was this: 1. shift transactionDate into processed_at 2. override processed_at and transform it into a date via toDate function 3. format it into my desired format via format function

This doesn't work, in the best case, I either get an empty processed_at or the initial value.

I tried

${processed_at.replaceAll('T',' '):toDate('yyyy-MM-dd HH:mm:ss'):format('yyyy-MM-dd HH:mm:ss')}

${processed_at:toDate('yyyy-MM-ddTHH:mm:ss'):format('yyyy-MM-dd HH:mm:ss')}

Answer 1

Apparently, I cannot access JSON properties with expression language in the jolt spec in the JoltTransformJSON processor.

The way I made it to work was:

1. I added before JoltTransformJSON an EvaluateJSONPath processor and extracted processed_at as a Flowfile attribute. My flow would look like this: EvaluateJSONPath -> JoltTransformJSON -> ConvertJsonToSql -> PutSql

2. In the JoltTransformJSON I now have access to the Flowfile attribute processed_at extracted earlier. In the Jolt spec, I updated the default operation: { "operation": "default", "spec": { "processed_at": null, "processed_at": "${processed_at:replace('T', ''):toDate('yyyy-MM-ddHH:mm:ss'):format('yyyy-MM-dd HH:mm:ss.SSS')}" } }

3. The correct SQL date field format in expression language is: yyyy-MM-dd HH:mm:ss.SSS

Now the flow inserts rows into the database.