spark-submit 傳遞 --files

Question

我想在 spark-submit 中傳遞兩個文件：

• 密鑰.jks
• 信任庫

位置如下：

• /home/my_user/config/key.jks
• /home/my_user/config/trustore.jks

我執行以下命令：

spark-submit --name historization \
--class com.ConsumerLauncher \
--master yarn \
--deploy-mode cluster \
--files /home/my_user/config/key.jks,/home/my_user/config/trustore.jks \
/home/my_user/jars/app.jar

我想在我的代碼中為我的 kafka 參數處理這兩個文件：

val kafkaParams = Map[String, Object](
  "bootstrap.servers" -> "myhost",
  "group.id" -> "grp-test",
  "key.deserializer" -> classOf[StringDeserializer],
  "value.deserializer" -> classOf[StringDeserializer],
  "auto.offset.reset" -> "earliest",
  "enable.auto.commit" -> (false: java.lang.Boolean),
  "security.protocol" -> "SSL",
  "ssl.truststore.password" -> "xxxx",
  "ssl.key.password" -> "xxxx",
  "ssl.keystore.password" -> "xxxx",
  //this two lines
  "ssl.truststore.location" -> "/home/my_user/config/trustore.jks",
  "ssl.keystore.location" -> "/home/my_user/config/key.jks"
  //this two line also bug
  //"ssl.truststore.location" -> getClass.getResource("/ssl/trustore.jks").getPath,
  //"ssl.keystore.location" -> getClass.getResource("/ssl/key.jks").getPath
)

不幸的是我收到了這個錯誤

Caused by: java.io.FileNotFoundException: /home/my_user/config/key.jks (No such file or directory)

你有什么主意嗎？

Answer 1

當您使用 --files 時，您的文件將上傳到 HDFS。 您現在可以通過以下方式訪問：trustore.jks 和 key.jks

val kafkaParams = Map[String, Object](
  "bootstrap.servers" -> "myhost",
  "group.id" -> "grp-test",
  "key.deserializer" -> classOf[StringDeserializer],
  "value.deserializer" -> classOf[StringDeserializer],
  "auto.offset.reset" -> "earliest",
  "enable.auto.commit" -> (false: java.lang.Boolean),
  "security.protocol" -> "SSL",
  "ssl.truststore.password" -> "xxxx",
  "ssl.key.password" -> "xxxx",
  "ssl.keystore.password" -> "xxxx",
  //this two lines
  "ssl.truststore.location" -> "trustore.jks",
  "ssl.keystore.location" -> "key.jks"
  //this two line also bug
  //"ssl.truststore.location" -> getClass.getResource("/ssl/trustore.jks").getPath,
  //"ssl.keystore.location" -> getClass.getResource("/ssl/key.jks").getPath
)