使用hadoop FileSystem从本地文件系统中的jar文件读取

Question

We have a maven project with some files in the resources dir which get copied into the root of the jar file. I have the following bit of code which works fine during JUnit testing but stops working once I try to execute it from the jar

        Configuration configuration = new Configuration();
        String pathString = MainClass.class.getClassLoader().getResource("dir").getPath();
        Path path = new Path(pathString);

        logger.debug(path);
        FileSystem fs = path.getFileSystem(configuration);
        if (fs.exists(path)) {
            logger.debug("WOOOOO");
        } else {
            logger.debug("BOOOOO");
        }

While testing, the output is:

DEBUG: /path/to/project/target/test-classes/dir
DEBUG: WOOOOO

While running from jar I get:

DEBUG file:/path/to/jar/project.jar!/dir
DEBUG BOOOOO

Needless to say, the jar file is in the correct location and the dir is in the root of that jar.

In case you're wondering why we're doing this, the second half is little test excerpt, which mimics what NaiveBayesModel.materialize() in Mahout does. We just need to be able to create a path that Mahout will understand.

Answer 1

The exception java.io.IOException: No FileSystem for scheme: jar means that you can't create a File object or open an FSDataInputStream (What Mahout does) with an URI that references something inside a jar object.

Schemes file and hdfs have FileSystem implementations, hence, I guess the only solution for you case, since you want to call NaiveBayesModel.materialize() , is to dump the files inside the dir directory of your jar into one of the two FileSystem that I mentioned and then create a Path from it.

In other hand, you can try to reproduce what Mahout does , which is the instantiation of a NaiveBayesModel .

I don't have experience with Mahout, but I guess it's a good point to start, hope it helps.