我正在嘗試使用 Springboot 應用程序為 AWS MSK 創建一個生產者，能夠從 EC2 客戶端創建它（使用 kafka-console-producer.sh）

Question

在向 msk(kafka 2.1.0) 生成消息時我得到

“向主題 AWSKafkaTopic 發送帶有key='null'和payload='Message->0'的消息時拋出異常”

我正在嘗試使用 docker 從部署在 EC2 上的 springboot 應用程序生成它。但是當我嘗試使用 kafka-console-producer.sh 從同一個 EC2 客戶端生成消息時，生產者工作正常。

bin/kafka-console-producer.sh --broker-list "XXBootstrapBrokerStringTlsXX" --producer.config client.properties --topic AWSKafkaTopic

我在本地使用 kafka 2.3.0 和 zookeeper 嘗試了相同的程序，它在那里工作正常（在 docker 上運行 springboot 應用程序）。

Config-> 
    @Value("${spring.kafka.producer.bootstrap-servers}")
        private String bootstrapServers;

        @Bean
        public Map<String, Object> producerConfigs() {
            Map<String, Object> props = new HashMap<>();
            props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,bootstrapServers);    props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,StringSerializer.class);        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,StringSerializer.class);
            return props;
        }

        @Bean
        public ProducerFactory<String, String> producerFactory() {
            return new DefaultKafkaProducerFactory<>(producerConfigs());
        }

        @Bean
        public KafkaTemplate<String, String> kafkaTemplate() {
            return new KafkaTemplate<>(producerFactory());
        }

        @Bean
        public Sender sender() {
            return new Sender();
        }


Client->
     @Autowired
        private KafkaTemplate<String,String> kafkaTemplate;

        public void sendMessage(String message){
            this.kafkaTemplate.send("AWSKafkaTopic",message);
        }

Actual result->

    ProducerConfig values:
            acks = 1
            batch.size = 16384
            bootstrap.servers = [XXBootstrapBrokerStringTlsXX]
            buffer.memory = 33554432
            client.id =
            compression.type = none
            connections.max.idle.ms = 540000
            enable.idempotence = false
            interceptor.classes = []
            key.serializer = class org.apache.kafka.common.serialization.StringSerializer
            linger.ms = 0
            max.block.ms = 60000
            max.in.flight.requests.per.connection = 5
            max.request.size = 1048576
            metadata.max.age.ms = 300000
            metric.reporters = []
            metrics.num.samples = 2
            metrics.recording.level = INFO
            metrics.sample.window.ms = 30000
            partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
            receive.buffer.bytes = 32768
            reconnect.backoff.max.ms = 1000
            reconnect.backoff.ms = 50
            request.timeout.ms = 30000
            retries = 0
            retry.backoff.ms = 100
            sasl.client.callback.handler.class = null
            sasl.jaas.config = null
            sasl.kerberos.kinit.cmd = /usr/bin/kinit
            sasl.kerberos.min.time.before.relogin = 60000
            sasl.kerberos.service.name = null
            sasl.kerberos.ticket.renew.jitter = 0.05
            sasl.kerberos.ticket.renew.window.factor = 0.8
            sasl.login.callback.handler.class = null
            sasl.login.class = null
            sasl.login.refresh.buffer.seconds = 300
            sasl.login.refresh.min.period.seconds = 60
            sasl.login.refresh.window.factor = 0.8
            sasl.login.refresh.window.jitter = 0.05
            sasl.mechanism = GSSAPI
            security.protocol = PLAINTEXT
            send.buffer.bytes = 131072
            ssl.cipher.suites = null
            ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
            ssl.endpoint.identification.algorithm = https
            ssl.key.password = null
            ssl.keymanager.algorithm = SunX509
            ssl.keystore.location = null
            ssl.keystore.password = null
            ssl.keystore.type = JKS
            ssl.protocol = TLS
            ssl.provider = null
            ssl.secure.random.implementation = null
            ssl.trustmanager.algorithm = PKIX
            ssl.truststore.location = null
            ssl.truststore.password = null
            ssl.truststore.type = JKS
            transaction.timeout.ms = 60000
            transactional.id = null
            value.serializer = class
value.serializer = class org.apache.kafka.common.serialization.StringSerializer

日志：

2019-07-24 07:40:43.305  INFO 1 --- [nio-9000-exec-1] o.a.kafka.common.utils.AppInfoParser     : Kafka version : 2.0.1
2019-07-24 07:40:43.305  INFO 1 --- [nio-9000-exec-1] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId : fa14705e51bd2ce5
2019-07-24 07:41:43.313 ERROR 1 --- [nio-9000-exec-1] o.s.k.support.LoggingProducerListener    : Exception thrown when sending a message with key='null' and payload='Message->0' to topic AWSKafkaTopic:
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

Answer 1

就我而言，我嘗試在新主題中生成消息，但auto.create.topics.enable在 aws 代理中為false ，因此最好在現有主題上創建消息或auto.create.topics.enable將此屬性設置為true並嘗試.