在分布式模式下运行kafka connect时出现问题
[英]Issues while running kafka connect in distributed mode
问题描述
We are testing kafka connect in distributed mode to pull topic records from kafka to HDFS. 
我们正在测试kafka connect在分布式模式下将主题记录从kafka拉到HDFS。 We have two boxes. 我们有两个盒子。 One in which kafka and zookeeper daemons are running. 其中一个运行kafka和zookeeper守护程序。 We have kept one instance of kafka connect in this box. 我们在此框中保留了一个kafka connect实例。 We have another box where HDFS namenode is present. 我们还有一个盒子，其中存在HDFS namenode。 We have kept another instance of kafka connect here. 我们在这里保留了另一个kafka实例。
We started kafka,zookeeper and kafka connect in first box. 我们在第一个盒子中启动了kafka，zookeeper和kafka connect。 We started kafka connect in second box as well. 我们也在第二个盒子中开始了kafka connect。 Now as per confluent documentation, we have to start the HDFS connector(or any other connector for that matter) using REST API. 现在，根据融合的文档，我们必须使用REST API启动HDFS连接器（或与此相关的任何其他连接器）。 So, after starting kafka connect in these two boxes, we tried starting connector through REST API. 因此，在这两个框中启动kafka connect之后，我们尝试通过REST API启动连接器。 We tried below command:- 我们尝试了以下命令：
curl -X POST -H "HTTP/1.1 Host: ip-10-16-34-57.ec2.internal:9092 Content-Type: application/json Accept: application/json" --data '{"name": "hdfs-sink", "config": {"connector.class":"io.confluent.connect.hdfs.HdfsSinkConnector", "format.class":"com.qubole.streamx.SourceFormat", "tasks.max":"1", "hdfs.url":"hdfs://ip-10-16-37-124:9000", "topics":"Prd_IN_TripAnalysis,Prd_IN_Alerts,Prd_IN_GeneralEvents", "partitioner.class":"io.confluent.connect.hdfs.partitioner.DailyPartitioner", "locale":"", "timezone":"Asia/Calcutta" }}' http://ip-10-16-34-57.ec2.internal:8083/connectors
As soon as we press enter here, we get below response: 在此处按Enter键后，我们将收到以下响应：
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 415 </title>
</head>
<body>
<h2>HTTP ERROR: 415</h2>
<p>Problem accessing /connectors. Reason:
<pre> Unsupported Media Type</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
</body>
</html>
The connect-distributed.properties file at etc/kafka/ is below in both the kafka connect nodes. 两个kafka连接节点中的etc / kafka /下的connect-distributed.properties文件。 We have created the said three topics as well (connect-offsets,connect-configs,connect-status) 我们还创建了上述三个主题（connect-offsets，connect-configs，connect-status）
bootstrap.servers=ip-10-16-34-57.ec2.internal:9092
group.id=connect-cluster
key.converter=com.qubole.streamx.ByteArrayConverter
value.converter=com.qubole.streamx.ByteArrayConverter
enable.auto.commit=true
auto.commit.interval.ms=1000
offset.flush.interval.ms=1000
key.converter.schemas.enable=true
value.converter.schemas.enable=true
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
offset.storage.topic=connect-offsets
rest.port=8083
config.storage.topic=connect-configs
status.storage.topic=connect-status
offset.flush.interval.ms=10000
What is the issue here? 这是什么问题？ Are we missing something to start kafka connect in distributed mode to work with HDFS connectors. 我们是否缺少启动分布式模式下的kafka connect以便与HDFS连接器一起使用的东西？ kafka connect in standalone mode is working fine. 独立模式下的kafka connect正常工作。
1 个解决方案
解决方案1
0 2017-02-02 07:02:04
To upload a connector, this is a PUT command, not a POST: http://docs.confluent.io/3.1.1/connect/restapi.html#put--connectors-(string-name)-config 要上传连接器，这是一个PUT命令，而不是POST： http : //docs.confluent.io/3.1.1/connect/restapi.html#put--connectors-(string-name)-config
On a side note, I believe that you curl command might be wrong: 附带一提，我相信您curl命令可能是错误的：
- you need one -H switch per header, putting all headers in one -H parameter is not how it works (I think). 您需要每个标头一个-H开关，将所有标头放在一个-H参数中是不起作用的（我认为）。
- I do not think that the port is part of the Host header. 我认为该端口不是Host标头的一部分。
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.