简体繁体 English

Kafka Connect分隔日志记录

[英]Kafka Connect separated logging

原文 2018-09-06 09:15:39 3 1 debugging/ logging/ apache-kafka/ apache-kafka-connect

Currently we are using a couple of custom connetor plugins for our confluent kafka connect distributed worker cluster. 当前，我们为融合的kafka connect分布式工作者集群使用了两个自定义的connetor插件。 One thing that bothers me for a long time is that kafka connect writes all logs from all deployed connectors to one File/Stream. 很长时间困扰我的一件事是，kafka connect将所有已部署的连接器中的所有日志写入一个文件/流。 This makes debugging an absolute nightmare. 这使得调试绝对是一场噩梦。 Is there a way to let kafka connect log the connectors in different Files/Streams? 有没有办法让kafka连接记录不同文件/流中的连接器？

Via the connect-log4j.properties I am able to let a specific class log to a different File/Stream. 通过connect-log4j.properties我可以让特定的类记录到其他File / Stream。 But this means that with every additional connector I have to adjust the connect-log4j.properties 但这意味着每个其他连接器我都必须调整connect-log4j.properties

Thanks 谢谢

1 个解决方案

Kafka Connect does not currently support this. Kafka Connect当前不支持此功能。 I agree that it is not ideal. 我同意这是不理想的。

One option would be to split out your connectors and have a dedicated worker cluster for each, and thus separate log files. 一种选择是拆分您的连接器，并为每个连接器分配一个专用的工作程序群集，从而使日志文件分离。

Kafka Connect is part of Apache Kafka so you could raise a JIRA to discuss this further and maybe contribute it back via a PR? Kafka Connect是Apache Kafka的一部分，因此您可以提出JIRA进行进一步讨论，也可以通过PR进行回馈吗？

Edit April 12, 2019: See https://cwiki.apache.org/confluence/display/KAFKA/KIP-449%3A+Add+connector+contexts+to+Connect+worker+logs 编辑2019年4月12日：请参阅https://cwiki.apache.org/confluence/display/KAFKA/KIP-449%3A+Add+connector+contexts+to+Connect+worker+logs