[英]Unable To Read Messages From Kafka Topic Using Spark Streaming Kafka
[英]Unable to read messages from kafka topic in ksqldb
{
"event": {
"header":{
"name":"abc",
"version":"1.0",
"producer":"123",
"channel":"lab",
"countryCode":"US"
},
"body":{
"customerIdentifiers":[
{"customerIdentifier":"1234","customerIdType":"cc"},
{"customerIdentifier":"234", "customerIdType":"id"}
],
"accountIdentifiers":[
{"accountIdentifier":"123", "accountIdType":"no"},
{"accountIdentifier":"Primary","accountIdType":"da"}
],
"eventDetails":{
"transactionDateTime":"2019-03-26 05:28:13.000",
"transactionDate":"2019-03-26",
"monthAverage":"188",
"dailyAverage":"7"
}
}
}
}
為上述 json 創建了 stream:
CREATE STREAM STREAM_NAME(
event STRUCT<
header STRUCT<
name VARCHAR,
version VARCHAR,
producer VARCHAR,
channel VARCHAR,
countryCode VARCHAR
eventTimeStamp VARCHAR
>,
body STRUCT<
customerIdentifiers STRUCT<
customerIdentifier VARCHAR,
customerIdType VARCHAR
>,
accountIdentifiers STRUCT<
accountIdentifier VARCHAR,
accountIdType VARCHAR
>,
eventDetails STRUCT<
transactionDateTime VARCHAR,
transactionDate VARCHAR,
productDescription VARCHAR,
monthAverage VARCHAR,
dailyAverage VARCHAR
>
>
>
) WITH (
KAFKA_TOPIC = 'TOPIC1',
VALUE_FORMAT = 'JSON',
PARTITIONS = 1
;
我無法讀取來自 stream 的消息:
select * from STREAM_NAME emit changes;
請問有什么建議嗎?
您可能會遇到反序列化錯誤,因為您創建的 stream 的架構與數據的架構不匹配。
重新格式化示例數據和 SQL 后,錯誤變得更加明顯:
$.event.body.customerIdentifiers
元素是一個結構數組,但 DDL 將其定義為一個結構。$.event.body.accountIdentifiers
元素是一個結構數組,但 DDL 將其定義為一個結構。應該工作的 DDL 是:
CREATE STREAM STREAM_NAME(
event STRUCT<
header STRUCT<
name VARCHAR,
version VARCHAR,
producer VARCHAR,
channel VARCHAR,
countryCode VARCHAR
eventTimeStamp VARCHAR
>,
body STRUCT<
customerIdentifiers ARRAY<STRUCT<
customerIdentifier VARCHAR,
customerIdType VARCHAR
>>,
accountIdentifiers ARRAY<STRUCT<
accountIdentifier VARCHAR,
accountIdType VARCHAR
>>,
eventDetails STRUCT<
transactionDateTime VARCHAR,
transactionDate VARCHAR,
productDescription VARCHAR,
monthAverage VARCHAR,
dailyAverage VARCHAR
>
>
>
) WITH (
KAFKA_TOPIC = 'TOPIC1',
VALUE_FORMAT = 'JSON',
PARTITIONS = 1
;
另請注意,數據不包含 DDL 中的以下內容(盡管這不會導致任何問題,因為字段將只是NULL
):
$.event.header.eventTimeStamp
$.event.body.eventDetails.productDescription
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.