为什么 HDFS FileNotFoundException：文件不存在？

Question

I visit HDFS from K8S pod, but it could not find the HDFS file.

Exception in thread "main" java.io.FileNotFoundException: File does not exist: hdfs://192.168.65.2:8020/user/flink/[/user/flink/.flink/job-1234/1624538371951/systemShipFiles/examples-wc.jar]
    at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
    at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
    at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2030)
    at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1999)
    at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1975)
    at org.apache.flink.kubernetes.entrypoint.KubernetesInitContainerEntrypoint.fetchRemoteDependencies(KubernetesInitContainerEntrypoint.java:121)
    at org.apache.flink.kubernetes.entrypoint.KubernetesInitContainerEntrypoint.main(KubernetesInitContainerEntrypoint.java:72)
Error from server (BadRequest): container "flink-job-manager" in pod "job-1234-b8d68f956-c4g58" is waiting to start: PodInitializing

But I can got the file with the hadoop command as flows:

hadoop fs -ls /user/flink/.flink/job-1234/1624538371951/systemShipFiles/oceanus-examples-wc.jar
21/06/24 20:43:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
-rw-r--r--   1 flink supergroup      13981 2021-06-24 20:39 /user/flink/.flink/job-1234/1624538371951/systemShipFiles/examples-wc.jar

The HADOOP is in local mode.

Thanks for your help.

Answer 1

Can you try ls with full path that is prefixed with hdfs://192.168.65.2:8020/user/...... as it is as it appears in the stacktrace.

Check in the core-site.xml, the fs.defaultFs value is specified and correct.