How to set hadoop replication in java client by class org.apache.hadoop.conf.Configuration?

Question

I use the java Api as client to upload files ,but it always set the dfs.replication to 3,as a result when I use command (hadoop dfsadmin -report) to check the situation,all blocks are under replication factor,because I just have two data nodes to test.

I just want to know how to set hadoop dfs.replication in java client by class org.apache.hadoop.conf.Configuration or in another way? Thank you for your help!

Answer 1

IF file is already copied to HDFS with 3 blocks replication, you can use setReplication API along with your file src and expected replication count to set replication specific to your file as below:

FileSystem fs = FileSystem.get(new Configuration());
fs.setReplication(new Path("hdfs_path:/foldername/filename"), (short)2);

setReplication API is described here : public boolean setReplication(Path src, short replication)

Also if want to copy files with 2 blocks replication factor you can pass replication count directly in your command as below:

hadoop fs -D dfs.replication=2 YOUR_COMMANDS

Answer 2

According to the Java API for the Hadoop FileSystem class you can specify the replication factor for a file when creating the output stream to write a file to the cluster. Eg

create(Path f, short replication) 

I currently cannot test this locally as I only have a Zookeeper node running here.