简体   繁体   English

使用 kafka-json-schema-console-consumer 从 KAFKA 消费键/值仅返回值而不返回键

[英]Consuming a key/value from KAFKA using kafka-json-schema-console-consumer only returns value but not key

I am building a Kafka source connector and am able to successfully publish a Kafka SourceRecord with a key and its schema with a value and its schema.我正在构建一个 Kafka 源连接器,并且能够成功发布带有键的 Kafka SourceRecord 及其带有值及其架构的架构。 However, when I use kafka-json-schema-console-consumer to consume the message on the topic I expect both key and value to be received, but I only receive the value and the key is NOT received.但是,当我使用 kafka-json-schema-console-consumer 来消费有关主题的消息时,我希望收到键和值,但我只收到值而没有收到键。 Any help is appreciated!任何帮助表示赞赏!

Below is the relevant code that I have tried/used:以下是我尝试/使用的相关代码:

The message that I am publishing:我发布的消息:

   "message": {
        "key": {
            "id": "some-id"
        },
        "data": {
            "text": "some-text"
        }
    }

The message gets put into he SourceRecord as follows:消息按如下方式放入他的 SourceRecord:

:
:
SourceRecord sr = new SourceRecord(
            sourcePartition, sourceOffset, this.topicName, partition,
            keySchema, key, valueSchema, value);
log.info("HttpSourceTask:buildSourceRecord - source:{}", sr);

I use a log message to confirm the message was successfully processed... below is the console log message in Kafka which confirms that the key plus key schema and the value plus value schema are correctly set in the SourceRecord:我使用日志消息来确认消息已成功处理...下面是 Kafka 中的控制台日志消息,它确认在 SourceRecord 中正确设置了键加键模式和值加值模式:

connect            | [2022-01-29 21:24:47,455] INFO HttpSourceTask:buildSourceRecord - source:SourceRecord{sourcePartition={source=bgs-topic}, sourceOffset={offset=0}} ConnectRecord{topic='bgs-topic', kafkaPartition=null, key=Struct{id=some-id}, keySchema=Schema{STRUCT}, value=Struct{text=some-text}, valueSchema=Schema{STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)}

Note that the key and keySchema fields are filled in with a STRUCT (as are the value and valueSchema fields).请注意,key 和 keySchema 字段是用 STRUCT 填充的(value 和 valueSchema 字段也是如此)。

I am using the following code (I am using Kafka in docker) to consume the message... this is the part that DOES NOT seem to return the right data (key is missing).我正在使用以下代码(我在 docker 中使用 Kafka)来使用消息......这是似乎没有返回正确数据的部分(缺少密钥)。

docker exec -it schema-registry \
/usr/bin/kafka-json-schema-console-consumer \
--bootstrap-server http://kafka:9092 \
--topic bgs-topic \
--from-beginning \
--property key.separator="|" \
--property value.schema='
{
  "title": "simple-data-schema",
  "type" : "object",
  "required" : [ "text" ],
  "properties" : {
    "text" : {
      "type": "string"
    }
  }
}' \
--property parse.key=true \
--property key.schema='
{
  "title": "simple-key-schema",
  "type" : "object",
  "required" : [ "id" ],
  "properties" : {
    "id" : {
      "type": "string"
    }
  }
}'

Here is the returned message - note that the only value is available:这是返回的消息 - 请注意,唯一的值是可用的:

{"text":"some-text"}

Found the error... I used "parse.key=true" but it should be "print.key=true"... once this change was made the correct output was provided:发现错误...我使用了“parse.key=true”,但它应该是“print.key=true”...一旦进行此更改,就会提供正确的 output:

{"id":"some-id"}|{"text":"some-text"}

The corrected command is shown below:更正后的命令如下所示:

docker exec -it schema-registry docker exec -it schema-registry
/usr/bin/kafka-json-schema-console-consumer /usr/bin/kafka-json-schema-console-consumer
--bootstrap-server http://kafka:9092 --bootstrap-server http://kafka:9092
--topic bgs-topic --topic bgs-topic
--from-beginning - 从开始
--property key.separator="|" --property key.separator="|"
--property value.schema=' { "title": "simple-data-schema", "type": "object", "required": [ "text" ], "properties": { "text": { "type": "string" } } }' --property value.schema=' { "title": "simple-data-schema", "type": "object", "required": [ "text" ], "properties": { "text": { "类型”:“字符串”} } }'
--property print.key=true --property print.key=true
--property key.schema=' { "title": "simple-key-schema", "type": "object", "required": [ "id" ], "properties": { "id": { "type": "string" } } }' --property key.schema=' { "title": "simple-key-schema", "type": "object", "required": [ "id" ], "properties": { "id": { "类型”:“字符串”} } }'

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM