[英]Kafka schema registry RestClientException: Unauthorized; error code: 401
I am trying to read data from a kafka avro topic using the avro schema from the confluent client registry.我正在尝试使用融合客户端注册表中的 avro 模式从 kafka avro 主题读取数据。 I am using io.confluent
library version 5.4.1
.我正在使用io.confluent
库版本5.4.1
。 This is the entry in the gradle file这是 gradle 文件中的条目
compile (group: 'io.confluent', name: 'kafka-avro-serializer', version: '5.4.1') {
exclude group: 'org.apache.avro', module: 'avro'
}
I receive the following error.我收到以下错误。
Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Unauthorized; error code: 401
public PCollection<KV<String, GenericRecord>> apply(Pipeline p) {
ConfluentSchemaRegistryDeserializerProvider<GenericRecord> valDeserializerProvider =
ConfluentSchemaRegistryDeserializerProvider.of(params.schemaUrl, "topic-value");
PCollection<KafkaRecord<String, GenericRecord>> records = p.apply("GetDataFromKafka", KafkaIO.<String, GenericRecord>read()
.withBootstrapServers(params.apiHost)
.withTopics("topic")
.withConsumerConfigUpdates(params.getConsumerProps())
.withKeyDeserializer(StringDeserializer.class)
.withValueDeserializer(valDeserializerProvider)
.commitOffsetsInFinalize());
return records.apply("TopicAndDataInput", MapElements.via(new SimpleFunction<KafkaRecord<String, GenericRecord>, KV<String, GenericRecord>>() {
@Override
public KV<String, GenericRecord> apply(KafkaRecord<String, GenericRecord> input) {
String topic = input.getTopic();
GenericRecord data = input.getKV().getValue();
return KV.of(topic, data);
}
}));
}
What am I missing here?我在这里错过了什么? Could someone point me in the right direction.有人能指出我正确的方向吗? Thanks in advance.提前致谢。
This is the function to get consumer properties这是function获取消费属性
public Map<String, Object> getConsumerProps() {
Map<String, Object> props = new HashMap<> ();
props.put("group.id", groupId);
props.put("auto.offset.reset", "earliest");
props.put("retry.backoff.ms",500);
props.put("max.partition.fetch.bytes", 8388608);
props.put("basic.auth.credentials.source", "USER_INFO");
props.put("basic.auth.user.info", "registry_key:secret");
props.put("ssl.endpoint.identification.algorithm","https");
props.put("security.protocol","SASL_SSL");
props.put("sasl.jaas.config","org.apache.kafka.common.security.plain.PlainLoginModule required username='"+ apiKey +"' password='" + apiSecret +"';");
props.put("sasl.mechanism","PLAIN");
return props;
}
Tried also with the following props and still get the same unauthorized error.还尝试了以下道具,但仍然遇到相同的未经授权的错误。
props.put("basic.auth.credentials.source", "USER_INFO");
props.put("schema.registry.basic.auth.user.info", "<registry key>:<value>");
props.put("schema.registry.url", schemaUrl);
I used to have the same issue with libraries 5.3.0
.我曾经对库5.3.0
有同样的问题。 I resolved it updating to我解决了它更新到
'org.apache.avro:avro:1.10.2'
'io.confluent:kafka-schema-registry-client:6.2.0'
I am using the following props to connect Schema Registry:我正在使用以下道具连接架构注册表:
"schema.registry.url": "<URL>"
"schema.registry.basic.auth.credentials.source": "USER_INFO"
"schema.registry.basic.auth.user.info": "<API_KEY>:<API_SECRET>"
Seems like you should add the specific schema registry key and secret as似乎您应该将特定的模式注册表项和机密添加为
props.put("schema.registry.basic.auth.user.info", "<SCHEMA_REGISTRY_API_KEY>:<SCHEMA_REGISTRY_API_SECRET>");
to the properties.到属性。 (from https://docs.confluent.io/cloud/current/cp-component/streams-cloud-config.html ) (来自https://docs.confluent.io/cloud/current/cp-component/streams-cloud-config.html )
Be sure to use at least version 5.0 of the Confluent schema registry libraries kafka-schema-registry
& kafka-avro-serializer
, as earlier versions don't support authentication.请务必使用至少 5.0 版的 Confluent 模式注册表库kafka-schema-registry
和kafka-avro-serializer
,因为早期版本不支持身份验证。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.