az 突触火花作业提交

Question

According to the documentation, using az synapse spark job submit, I can pass arguments using --arguments. So far so good.
However, I cannot figure out to actually access those arguments in my code. Here's my current effort:

val conf = new SparkConf().setAppName("foo")
val sc = new SparkContext(conf)
val spark = SparkSession.builder.appName("foo").getOrCreate()
val start_time = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm").format(LocalDateTime.now)
val appID = sc.getConf.getAppId


//let's get some arguments
val inputArgs = spark.sqlContext.getConf("spark.driver.args").split("\\s+")
//val inputArgs = sc.getConf.get("spark.driver.args").split("\\s+") 

Either of those lines throw the following exception:

22/03/25 19:07:45 ERROR ApplicationMaster: User class threw exception: java.util.NoSuchElementException: spark.driver.args
java.util.NoSuchElementException: spark.driver.args

So, how do I read the arguments in the Scala code?

Answer 1

Ok, I was overcomplicating this.

def main(args: Array[String]) {
...

val foo = args(0)