Could not find file in CRD definition for Spark application

Question

I am using Spark operator with Kubernetes. I need a file fot the application execution. However, when I define it near my jar file in files section of the custom object definition I get an error which means the the file is not found:

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0
    at example.SparkExample$.main(SparkExample.scala:37)
    at example.SparkExample.main(SparkExample.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:846)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:194)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:921)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:932)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

This exception point to the SparkExample.scala:37 where I read class arguments:

  val dataFromFile = readFile(spark.sparkContext, args(0))

What is wrong with the custom object definition or arguments setup?

Here is how custom object definition looks like:

apiVersion: sparkoperator.k8s.io/v1alpha1
kind: SparkApplication
metadata:
  name: spark-example
  namespace: default
spec:
  type: Scala
  image: gcr.io/ynli-k8s/spark:v2.4.0-SNAPSHOT
  mainClass: example.SparkExample
  mainApplicationFile: http://ip:8089/spark_k8s_airflow.jar
  mode: cluster
  deps:
    files:
      - http://ip:8089/jar_test_data.txt
  driver:
    coreLimit: 1000m
    cores: 0.1
    labels:
      version: 2.4.0
    memory: 1024m
    serviceAccount: default
  executor:
    cores: 1
    instances: 1
    labels:
      version: 2.4.0
    memory: 1024m
  imagePullPolicy: Always

Answer 1

答案在此发出我刚才添加的参数给应用程序并更改我加载文件到星火应用程序的方式