Multiples Hadoop FileSystem实例

Question

I have a class (I removes try/catch for readability):

public class HadoopFileSystem {

    private FileSystem m_fileSystem = null;

    public HadoopFileSystem() {
        Configuration l_configuration = new Configuration();
        l_configuration .set("fs.default.name", "hdfs://localhost:9100");
        l_configuration .set("mapred.job.tracker", "localhost:9101");

        m_fileSystem = FileSystem.get(l_configuration );

    }

    public void close() {
        m_fileSystem.close();
    }

    public void createFile(String a_pathDFS) {
        m_fileSystem.create(new Path(a_pathDFS));
    }

}

In my program, I a first HadoopFileSysem object, I don't close it .

Then I create a second HadoopFileSysem object, and I close it.

Finally, when I want to use a function on m_fileSystem in my first object, I have the error : java.io.IOException: Filesystem closed

But I did'nt close it !

Here's a little code to illustrate my problem :

HadoopFileSystem h1 = new HadoopFileSystem();
HadoopFileSystem h2 = new HadoopFileSystem();

if(h1 == h2)
    System.out.println("=="); // No print
if(h1.equals(h2))
    System.out.println("equals"); // No print

h2.close();
h1.createFile("test.test"); // ERROR : java.io.IOException: Filesystem closed
h1.close();

Why ?

Answer 1

m_fileSystem = FileSystem.get(l_configuration ); is a static call even if you have two different objects created. You need to find a way to not make this call static for two different objects.

Try this out to do away with the problem,

conf.setBoolean("fs.hdfs.impl.disable.cache", true);

Answer 2

You are creating an object based on a configuration which is hard-coded. This basically means that you're creating 2 identical objects. Because these objects are identical, the JVM will reference to the same object. So h1 and h2 are referencing the same object.

The reason for this is because you're getting an existing instance of an object based on a configuration file. If the configuration is different for h1 and h2, it will not be the same instance anymore.