Java 中的 Spark 作业：如何在集群上运行时从“资源”访问文件

Question

I wrote a Spark job in Java. The job is packaged as a shaded jar and executed:

spark-submit my-jar.jar

In the code, there are some files (Freemarker templates) that reside in src/main/resources/templates . When run locally, I'm able access the files:

File[] files = new File("src/main/resources/templates/").listFiles();

When the job is run on a cluster, a null-pointer exception is returned when the previous line is executed.

If I run jar tf my-jar.jar I can see that the files are packaged in a templates/ folder:

 [...]
 templates/
 templates/my_template.ftl
 [...]

I'm just unable to read them; I suspect that .listFiles() tries to access the local filesystem on the cluster node, and the files aren't there.

I'm curious to know how I should package files to be used within a self-contained Spark job. I'd rather not copy them to HDFS outside of the job because it becomes messy to maintain.

Answer 1

Your existing code is referencing them as files which are not packaged up and shipped to the Spark nodes. But, since they're inside your jar file you should be able to reference them via Foo.getClass().getResourceAsStream("/templates/my_template_ftl") . More info on Java resource streams here: http://www.javaworld.com/article/2077352/java-se/smartly-load-your-properties.html

Answer 2

It appears that running Scala (2.11) code on Spark does not support accessing resources in shaded jars.

Executing this code:

var path = getClass.getResource(fileName)
println("#### Resource: " + path.getPath())

prints the expected string when run outside of Spark.

When run inside Spark, a java.lang.NullPointerException is raised because path is null.

Answer 3

I have accessed my resource file like below in spark-scala. I've share my code please check.

val fs=this.getClass().getClassLoader().getResourceAsStream("smoke_test/loadhadoop.txt")

val dataString=scala.io.Source.fromInputStream(fs).mkString