Kafka Hdfs 2 Sink 连接器无法在 hdfs 上写入

Question

Following is my kafka connector json file:

curl -s -k -X POST  http://cpnode.local.lan:8083/connectors -H "Content-Type: application/json" --data '{
"name":"jdbc-Hdfs2-Sink-Connector",
"config":{
"tasks.max":"1",
"batch.size":"1000",
"batch.max.rows":"1000",
"hdfs.poll.interval.ms":"500",
"connector.class":"io.confluent.connect.hdfs.HdfsSinkConnector",
"hdfs.url":"hdfs://hadoopnode.local.lan:9000",
"topics":"BookList2",
"flush.size":"1",
"confluent.topic.bootstrap.servers":"cpnode.local.lan:9092",
"confluent.topic.replication.factor":"1",
"value.converter":"io.confluent.connect.avro.AvroConverter",
"value.converter.schemas.enable":"true",
"value.converter.schema.registry.url":"http://cpnode.local.lan:8081",
"key.converter":"io.confluent.connect.avro.AvroConverter",
"key.converter.schemas.enable":"true",
"key.converter.schema.registry.url":"http://cpnode.local.lan:8081"
}
}' | jq '.'

When I try to use this connector I get following error:

{
  "name": "jdbc-Hdfs2-Sink-Connector",
  "connector": {
    "state": "RUNNING",
    "worker_id": "192.168.1.153:8083"
  },
  "tasks": [
    {
      "id": 0,
      "state": "FAILED",
      "worker_id": "192.168.1.153:8083",
      "trace": "org.apache.kafka.connect.errors.ConnectException: org.apache.hadoop.security.AccessControlException: Permission denied: user=cp-user, access=WRITE, inode=\"/\":hadoop:supergroup:drwxr-xr-x

I have tried export HADOOP_USER_NAME=hdfs and also hadoop configuration hdfs-site.xml

<property>
   <name>dfs.permissions</name>
   <value>false</value>
</property>

But I want a solution without compromising security.

cp-user is the name of my confluent platform user... Both the confluent and hdfs are on different VMs

Thanks in advance....

Answer 1

Your user: user= cp-user ,

Is trying to access=WRITE ,

To the location inode=\\"/\\"

Which has user/group ownership of hadoop:supergroup:drwxr-xr-x

---

Possible Solutions (non-overlapping):

1. Change cp-user to hadoop (I assume you are using the Docker container? If so, refer to the user directive of Docker . Otherwise, export HADOOP_USER_NAME=hadoop )
2. Create and add cp-user Unix account to the NameNodes of the Hadoop cluster and all datanodes
3. Use Kerberos