无法使用Windows bat文件启动kafka connect分发

Question

UPDATE:

I have taken a look at the kafka.jar mentioned - which seems to be added to the classpath by the CMD files that run the distributed worker - kafka-run-class.bat in particular and it is less than 1kb so doesn't seem to be a valid JAR - so maybe that first error is a red herring. But that aside if it runs on, the distributed worker just spits out a load of errors about ClassNotFoundException, then goes on to say every config value provided in my distributed properties config file is not recognised, and doesn't try and load any plugins

ORIGINAL question

I am simply trying a very simple test of starting a Kafka Connect distributed worker, no plugins yet - just trying to get the basic worker going.

My distributed config is pretty stock

bootstrap.servers=GBV04303950:9092,GBV04303950:9082
group.id=connect-cluster
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
offset.storage.topic=connect-offsets
offset.storage.replication.factor=2
config.storage.topic=connect-configs
config.storage.replication.factor=2
status.storage.topic=connect-status
status.storage.replication.factor=2

However I get the following error

C:\kafka\ClusterOne\connect-worker_1>.\bin\windows\connect-distributed.bat .\etc\kafka-connect-replicator\replicator-connect-distributed.properties
[2019-03-22 10:53:51,832] WARN could not create Dir using jarFile from url file:/C:/kafka/ClusterOne/connect-worker_1/share/java/kafka/kafka.jar. skip
ping. (org.reflections.Reflections)
java.util.zip.ZipException: error in opening zip file
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(Unknown Source)
        at java.util.zip.ZipFile.<init>(Unknown Source)
        at java.util.jar.JarFile.<init>(Unknown Source)
        at java.util.jar.JarFile.<init>(Unknown Source)
        at org.reflections.vfs.Vfs$DefaultUrlTypes$1.createDir(Vfs.java:216)
        at org.reflections.vfs.Vfs.fromURL(Vfs.java:99)
        at org.reflections.vfs.Vfs.fromURL(Vfs.java:91)
        at org.reflections.Reflections.scan(Reflections.java:240)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.scan(DelegatingClassLoader.java:412)
        at org.reflections.Reflections$1.run(Reflections.java:198)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

The only reason I can think of for this is my local drive is locked down and doesn't let you create folders unless you have Elevated rights. I am running this from a CMD prompt that was started with elevated rights but as I've noticed, this doesn't always translate down to passing those rights on to processes spawned from within the CMD prompt. But I'm not sure how I can provide elevated rights to the java process that's trying to unpack the JAR.

Also, that error is followed by a long stream of ClassNotFoundException, I assume because the above error means JARs cannot be unpacked to load all the required classes

eg

[2019-03-22 10:56:04,567] WARN could not get type for name org.apache.kafka.common.utils.MockTime from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name org.apache.kafka.common.utils.MockTime
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
        at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
        at org.reflections.Reflections.<init>(Reflections.java:126)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:404)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:304)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:242)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:190)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:183)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
        at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.utils.MockTime
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
        ... 10 more
[2019-03-22 10:56:04,706] WARN could not get type for name com.google.gson.JsonDeserializer from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name com.google.gson.JsonDeserializer
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
        at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
        at org.reflections.Reflections.<init>(Reflections.java:126)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:404)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:304)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:242)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:190)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:183)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
        at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: com.google.gson.JsonDeserializer
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
        ... 10 more
[2019-03-22 10:56:04,857] WARN could not get type for name org.scalatest.junit.JUnitSuite from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name org.scalatest.junit.JUnitSuite
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
        at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
        at org.reflections.Reflections.<init>(Reflections.java:126)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:404)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:304)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:242)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:190)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:183)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
        at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: org.scalatest.junit.JUnitSuite
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
        ... 10 more

Can anyone suggest if they think my theory about elevated rights is why this isn't working or how I could get around this?

Answer 1

Inside the share/java/kafka folder there is a kafka.jar about 26 bytes. If you look at it with a text editor, you can see, that it is a link. Copy the linked jar to kafka.jar.

On windows, you should also set the plugin.path to an absolute path. You will get a few errors, while starting, that is due to missing aws and other jars.