NiFi JoltTransformJson仅输出第一条记录

Question

I am sending to NiFi JoltTransformJSON a number of records like:

{"uid":"x1","timestamp_ms":1419717816375}
{"uid":"x2","timestamp_ms":1419717816376}
...

They are coming via ComsumeKafkaRecord, and its RecordWriter is JsonRecordSetWriter configured with "One line per object". I veried via Data provenance, that there are,say,5 records coming out of it when I send 5 to Kafka.

Jolt spec for Shift transform is

{
  "*": "&0",
 "timestamp_ms": "timestamp"
}

So I just want to rename one field. I verified in Advanced section that this spec changes my json as requested.

But when I start all components and send 5 messages, I only get the first from JoltTransformJson - verified with Data Provenance.

{"uid":"x1","timestamp":1419717816375}

What's wrong here?

Update: it turned out that I had to use JoltTransformRecord, then it went as I expected, ie 5 message in, 5 out.

But I am still puzzled what made JoltTransformJson behave like that?

Answer 1

What made it behave like that is the fact that you referred your data(one record per line) as JSON while your data is not JSON. If it was JSON, then it would at least start and end with [] and each line (except for the final one) end with a comma.

Since JoltTransformJSON accepts only JSON, it behaved like that.

Regarding your edit, JoltTransformRecord knows how to process the records individually, as different JSON objects, and that's why it worked using it.