Hadoop：使用Filesystem.append（）在Map阶段在HDFS中更新文件时进行同步？

Question

I'm wondering if Hadoop has any sort of synchronization protection when multiple nodes try accessing the same file on HDFS using the FileSystem append(Path p) method?

I append values to a text file in the mapper stages of my jobs and I was wondering what would happen if two mappers tried accessing the same file at the same time? I don't want them to overwrite each other, preferably I would like nodes to wait to gain access to the file until others are done with it so that only one node opens the file at any one time.

        BufferedWriter br=new BufferedWriter(new OutputStreamWriter(fs.append(new Path(tempFilePath))));
        br.append("value");
        br.close();

I know this isn't proper MapReduce but for some of the jobs I am running I have no other choice but to do this as I need to store some text values independent from the final output.

Answer 1

According to the FAQ :

HDFS supports exclusive writes only.

When the first client contacts the name-node to open the file for writing, the name-node grants a lease to the client to create this file. When the second client tries to open the same file for writing, the name-node will see that the lease for the file is already granted to another client, and will reject the open request for the second client .

Answer 2

I don't know fs.append is blocking or not, but in your case the best solution is using MultipleOutputs ( documentation ). This will allow you to write data independent from the final output.