无法使用控制台使用者从 Kafka 主题读取消息

Question

I created stream1 in KSQL (version 5.0 Beta) with a backing topic topic1 and avro schema. I am able to read all messages on topic1 using kafka-avro-console-consumer .

I then created stream2 in KSQL that's based on stream1 but with json format for the messages and backing topic named topic2 . I am able to read all the messages on topic2 using kafka-console-consumer

I created stream3 in KSQL based on stream2 with json message format and backing topic named topic3 . However, I am unable to read the messages on topic3 using kafka-console-consumer .

Using kafkacat I get offsets on various partitions on topic3 but none of the actual messages is being printed.

It looks likes the messages are in the topic but neither kafkacat not kafka-console-consumer is able to print it.

Tried using --from-beginning and --offset earliest --partition 0 with no luck.

Here are the KSQL statements

CREATE STREAM stream1(p_id STRING, location STRING, u_id STRING, r_id STRING, b_id STRING, recorded_dtm STRING, 
v_type STRING, value STRING) WITH (kafka_topic='topic1', value_format='AVRO');

CREATE STREAM stream2 WITH (KAFKA_topic='topic2', VALUE_FORMAT='json', TIMESTAMP='RECORDED_TIMESTAMP') 
AS select P_ID+'-'+LOCATION+'-'+U_ID+'-'+R_ID+'-'+B_ID+'-'+V_TYPE as PARTITION_KEY, 
LOCATION, U_ID, R_ID, V_TYPE, B_ID, STRINGTOTIMESTAMP(recorded_dtm, 'yyyyMMddHHmmss') as RECORDED_TIMESTAMP, 
P_ID, VALUE, RECORDED_DTM,'NM' as DATA_TYPE 
FROM stream1 PARTITION BY PARTITION_KEY;

CREATE STREAM stream3 WITH (KAFKA_topic='topic3', VALUE_FORMAT='json', TIMESTAMP='RECORDED_TIMESTAMP') 
AS select PARTITION_KEY, LOCATION, U_ID, R_ID, V_TYPE, B_ID, RECORDED_TIMESTAMP, 
P_ID, VALUE, RECORDED_DTM FROM stream2 PARTITION BY PARTITION_KEY;

Additional info

In ksql if I run SET 'auto.offset.reset'='earliest'; and run select * from stream1 limit 5; or select * from stream2 limit 5 I see records printed but select * from stream3 limit 5 does not return any records.

If I run describe extended stream3 I get

total-messages: 212

which happens to be the number of messages I sent to topic1

Answer 1

The root cause was the Timestamp on STREAM3 The value for recorded_dtm column sent to messages on topic1 was earlier than the log.retention.hours value set in the kafka server.properties .

Our log.retention.hours value is set to 24 hours and the recorded_dtm values were earlier than 24 hours. This caused the messages in STREAM3 and topic3 to be immediately removed based in the retention policy.