简体   繁体   中英

KSQLDB, create table from a kafka topic not working?

I have created a kafka topic, here's is the schema.

# topic_name: test.rf.partner

{
  "connect.name": "test.RF.partner.Value",
  "fields": [
    {
      "name": "partnerID",
      "type": "int"
    },
    {
      "name": "name",
      "type": "string"
    }
  ],
  "name": "Value",
  "namespace": "test.RF.partner",
  "type": "record"
}

I tried to create a table using,

CREATE TABLE partner (partnerID INTEGER PRIMARY KEY, name VARCHAR) WITH 
    (KAFKA_TOPIC = 'test.RF.partner',VALUE_FORMAT='AVRO');

Then when I do this the result I am getting is empty data, the data from topic is not going into the table,

SELECT * FROM partner EMIT CHANGES;

ksql> SELECT * FROM partner EMIT CHANGES;
+----------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
|PARTNERID                                                                                           |NAME                                                                                                |
+----------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+

Update:

ksql> PRINT 'rytfit.RF.partner' FROM BEGINNING;

Key format: JSON or HOPPING(KAFKA_STRING) or TUMBLING(KAFKA_STRING) or KAFKA_STRING
Value format: AVRO
rowtime: 2021/09/27 19:07:05.563 Z, key: {"partnerID":100}, value: {"partnerID": 100, "name": "kalai"}, partition: 0
rowtime: 2021/09/27 19:07:05.564 Z, key: {"partnerID":101}, value: {"partnerID": 101, "name": "kal"}, partition: 0
rowtime: 2021/09/27 19:07:05.564 Z, key: {"partnerID":102}, value: {"partnerID": 102, "name": "kalai"}, partition: 0
rowtime: 2021/09/27 19:07:05.564 Z, key: {"partnerID":103}, value: {"partnerID": 103, "name": "Intelles"}, partition: 0
rowtime: 2021/09/27 19:07:05.565 Z, key: {"partnerID":104}, value: {"partnerID": 104, "name": "kala"}, partition: 0
rowtime: 2021/09/27 19:07:05.565 Z, key: {"partnerID":105}, value: {"partnerID": 105, "name": "kala"}, partition: 0
rowtime: 2021/09/27 19:07:05.565 Z, key: {"partnerID":106}, value: {"partnerID": 106, "name": "kala"}, partition: 0
rowtime: 2021/09/27 19:07:05.566 Z, key: {"partnerID":107}, value: {"partnerID": 107, "name": "partner"}, partition: 0
rowtime: 2021/09/27 19:07:05.566 Z, key: {"partnerID":108}, value: {"partnerID": 108, "name": "part"}, partition: 0
rowtime: 2021/09/27 19:07:05.566 Z, key: {"partnerID":109}, value: {"partnerID": 109, "name": "par"}, partition: 0

It happens the same to me for any existing topic. However, a workaround is to create a stream first and then a materialized view with the aggregator method LATEST_BY_OFFSET

CREATE TABLE partners AS
SELECT  partnerID,
        LATEST_BY_OFFSET(name) AS name 
  FROM partners_stream
GROUP BY partnerID
EMIT CHANGES;

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM