[英]Can't start Kafka Connect with MongoDb plugin with Apache Kafka
I am new to Kafka and I want to see if I can sync MongoDb data with another system using Kafka.我是 Kafka 的新手,我想看看是否可以使用 Kafka 将 MongoDb 数据与另一个系统同步。
My set up:我的设置:
/usr/local/share/kafka/plugins
./usr/local/share/kafka/plugins
。./bin/connect-standalone.sh ./config/connect-standalone.properties /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/etc/MongoSourceConnector.properties
[2020-10-17 13:57:22,304] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:264)
[2020-10-17 13:57:22,305] INFO Added plugin 'com.mongodb.kafka.connect.MongoSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:193)
[2020-10-17 13:57:22,305] INFO Added plugin 'com.mongodb.kafka.connect.MongoSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:193)
Archive: mongodb-kafka-connect-mongodb-1.3.0.zip
creating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/
creating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/etc/
inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/etc/MongoSourceConnector.properties
inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/etc/MongoSinkConnector.properties
creating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/doc/
inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/doc/README.md
inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/doc/LICENSE.txt
inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/manifest.json
creating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/lib/
inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/lib/mongo-kafka-1.3.0-all.jar
creating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/assets/
inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/assets/mongodb-leaf.png
inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/assets/mongodb-logo.png
This plugin is from confluent page, I have also tried downloading it from Maven page.这个插件来自 confluent 页面,我也尝试从 Maven 页面下载它。 The problem is when I run Kafka Connect it fails because plugin is missing a Java dependency.
问题是当我运行 Kafka Connect 时它失败了,因为插件缺少 Java 依赖项。
[2020-10-17 13:57:24,898] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:121)
java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: org/apache/avro/Schema
at com.mongodb.kafka.connect.source.MongoSourceConfig.createConfigDef(MongoSourceConfig.java:591)
at com.mongodb.kafka.connect.source.MongoSourceConfig.<clinit>(MongoSourceConfig.java:293)
at com.mongodb.kafka.connect.MongoSourceConnector.config(MongoSourceConnector.java:91)
at org.apache.kafka.connect.connector.Connector.validate(Connector.java:129)
at com.mongodb.kafka.connect.MongoSourceConnector.validate(MongoSourceConnector.java:51)
at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:313)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:192)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:115)
Caused by: java.lang.NoClassDefFoundError: org/apache/avro/Schema
... 8 more
Caused by: java.lang.ClassNotFoundException: org.apache.avro.Schema
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:104)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 8 more
My impression was that plugin should look for dependencies in the jar file /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/lib/mongo-kafka-1.3.0-all.jar
not in the Java SDK.我的印象是插件应该在 jar 文件
/usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/lib/mongo-kafka-1.3.0-all.jar
查找依赖项而不是在 Java SDK 中。
What am I missing in this set up?我在这个设置中缺少什么?
A quick look at this should tell you if the error is correct...快速浏览一下应该会告诉您错误是否正确...
jar -tf mongo-kafka-1.3.0-all.jar | grep avro
If that JAR doesn't bundle Avro itself, then MSK very likely doesn't include Avro like Confluent Platform does (which I assume Mongo bundled their connector primarily for).如果该 JAR 不捆绑 Avro 本身,那么 MSK 很可能不会像 Confluent Platform 那样包括 Avro(我假设 Mongo 捆绑了他们的连接器主要是为了)。 At least, Avro is not a dependency of Apache Kafka, so that would explain that error.
至少,Avro 不是 Apache Kafka 的依赖项,因此可以解释该错误。
You will need to download the Avro JAR and place it on your Kafka Connect Classpath (or at least in that lib folder)您需要下载 Avro JAR 并将其放置在您的 Kafka Connect Classpath(或至少在该 lib 文件夹中)
I faced same issue, when running on my local.在本地运行时,我遇到了同样的问题。 I downloaded the jar(mongo-kafka-connect-1.6.0-confluent.jar) from confluent platform which does not provide uber jar anymore.
我从不再提供超级 jar 的融合平台下载了 jar(mongo-kafka-connect-1.6.0-confluent.jar)。 So I searched for uber jar and found below site from where, I could download uber jar(select all in Download dropdown) and that resolved the issue.
所以我搜索了 uber jar 并找到了下面的站点,我可以从那里下载 uber jar(在下载下拉列表中选择全部)并解决了问题。
https://search.maven.org/search?q=a:mongo-kafka-connect https://search.maven.org/search?q=a:mongo-kafka-connect
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.