Flink 无法访问 JAR 中的文件

Question

I tried to run a JAR in a flink cluster but I get this FileNotFound Exception.

Caused by: java.io.FileNotFoundException: File file:/tmp/flink-web-88bf3f41-94fc-40bd-a865-bb0e6d5ac95c/flink-web-upload/82227475-523d-4607-8ab2-09bae8602248-tutorial-1.0-jar-with-dependencies.jar!/ldbc_sample/edges.csv does not exist or the user running Flink ('userA') has insufficient permissions to access it.
    at org.apache.flink.core.fs.local.LocalFileSystem.getFileStatus(LocalFileSystem.java:106)

The csv files are located in a folder in the resources directory of the project.

I access the file path by:

URL resource = Helper.class.getClassLoader().getResource("ldbc_sample");
return resource.getPath();

I opened the jar and made sure that the files definitely exist, and I also run it locally, and it worked. What do I have to do, to make sure that flink can access my csv?

Answer 1

Maybe you want to pass your .csv as an argument to your program ? Something like:

def main(args: Array[String]): Unit = {
  val ldbcSample = ParameterTool.fromArgs(args).getRequired("ldbc_sample")
  ...
}

or you can make your .properties file with different arguments:

ldbc_sample: /ldbc_sample/edges.csv
topic_source: TOPIC_NAME

val jobParams = ParameterTool.fromArgs(args)
val jobArgs = ParameterTool.fromPropertiesFile(jobParams.getRequired("properties_file_path"))