Kafka Connect HDFS Sink与Azure Blob存储

Question

I want to connect to Azure Blob Storage with Kafka HDFS Sink Connector. So far I have done:

1. Set kafka-connect properties:

    hdfs.url=wasbs://<my_url> hadoop.conf.dir={hadoop_3_home}/etc/hadoop/ hadoop.home={hadoop_3_home} 

2. And in core-site.xml added support for wasbs:

    <property> <name>fs.wasbs.impl</name> <value>org.apache.hadoop.fs.azure.NativeAzureFileSystem</value> </property> 

3. Exported HADOOP_CLASSPATH variable, added to PATH

But anyway, Hadoop can not find the class - NativeAzureFileSystem :

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at io.confluent.connect.hdfs.storage.StorageFactory.createStorage(StorageFactory.java:29)
 ... 11 more
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem not found
 at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
 at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654)
 at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
 at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)

Could you please help with this issue. Is it even possible?

Answer 1

my goal is: backup everything from Kafka to Azure BLOB of any data format.

The HDFS and cloud connectors can't backup "any format". Confluent's Avro is the first class citizen of file formats. JSON secondly, but there is not a "plain text" format, from what I've found. I think the HDFS connector does support "byte array" format.

As I mentioned in the comments, in my opinion, a backup of Kafka is different than indefinitely retaining the data to a file system. Backing up Kafka-to-Kafka includes use of MirrorMaker.

If you want to use any format, Spark, Flink, NiFi, or Streamsets have more flexibility for handling that out of the box