HDFS Hadoop中的数据分发

Question

I configured 3 datanodes on my Linux machine. In my configuration, I configured the number of replication to be 1.

I am submitting a file to the hdfs, and found that the file has 3 copies on each datanodes (I checked it from the browser)

Isn't right that I should only see the file on 1 datanodes and on 1 replica?

Answer 1

Before going into the HDFS the file will be split into blocks and you should see one replica of each block on each datanode. The file as a whole won't be present on any of the datanode.

Please make sure that you have restarted HDFS daemons after changing the replication factor property in the hdfs-site.xml file.

Also It would be good if you can post your HDFS Console Snapshot.

Answer 2

I suspect that dfs.replication is set to 3 instead of 1

Make sure that below parameters are set to 1 in your hdfs-site.xml

dfs.replication : Default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time

dfs.namenode.replication.min : Minimal block replication.

Have a look at documentation for more details.