Is there any function for adding timestamp to our message before storing in cassandra using kafka-cassandra sink connector?

Question

I'm setting up my logserver. I'm forwarding logs using Fluentd to Kafka and then storing them in Cassandra for later use. For this I'm using kafka-cassandra sink connector. I have to store data chronologically for which I need to add timestamp to my messages in cassandra. How can this be done?

Datamountaineer connector uses kcql which i think doesn't support inserting timestamp to a log.

My connector configuration is as follows:

name=cassandra-sink
connector.class=com.datamountaineer.streamreactor.connect.cassandra.sink.CassandraSinkConnector
tasks.max=1
topics=test_AF1
connect.cassandra.kcql=INSERT INTO test_event1 SELECT now() as id, message as msg FROM test_AF1 TIMESTAMP=sys_time()
connect.cassandra.port=9042
connect.cassandra.contact.points=localhost
connect.cassandra.key.space=demo

Answer 1

Kafka Connect's Single Message Transform can do this. Here's an example:

{
  "connector.class": "com.datamountaineer.streamreactor.connect.cassandra.sink.CassandraSinkConnector",
  "topics": "test_AF1",
…
  "transforms": "addTS",
  "transforms.addTS.type": "org.apache.kafka.connect.transforms.InsertField$Value",
  "transforms.addTS.timestamp.field": "op_ts"
}'

This adds a field to the message payload called op_ts with the timestamp of the Kafka message.

I don't know how this interacts with KCQL; you might want to check out the other two Cassandra sinks that I'm aware of:

• https://www.confluent.io/hub/confluentinc/kafka-connect-cassandra
• https://www.confluent.io/hub/datastax/kafka-connect-dse