简体繁体 English

在 emr 5.29.0 上运行 Scala 2.12

[英]Running scala 2.12 on emr 5.29.0

原文 2020-02-25 18:39:03 4 2 scala/ amazon-web-services/ apache-spark/ amazon-emr

I have a jar file compiled in scala 2.12 and now I want to run it on emr 5.29.0.我有一个在 Scala 2.12 中编译的 jar 文件，现在我想在 emr 5.29.0 上运行它。 How do I run them as the default version of emr 5.29.0 is scala 2.11.我如何运行它们，因为 emr 5.29.0 的默认版本是 scala 2.11。

2 个解决方案

As per this thread in AWS Forum , all Spark versions on EMR are built with Scala 2.11 as it's the stable version:根据AWS 论坛中的这个帖子，EMR 上的所有 Spark 版本都是使用 Scala 2.11 构建的，因为它是稳定版本：

On EMR, Spark is built with Scala-2.11.x, which is currently the stable version.在 EMR 上，Spark 使用 Scala-2.11.x 构建，这是目前的稳定版本。 As per- https://spark.apache.org/releases/spark-release-2-4-0.html , Scala-2.12 is still under experimental support.根据https://spark.apache.org/releases/spark-release-2-4-0.html ，Scala-2.12 仍在实验支持下。 Our service team is already aware of this feature request, and they shall be adding Scala-2.12.0 support in coming releases, once it becomes stable.我们的服务团队已经知道这个功能请求，一旦它变得稳定，他们将在即将发布的版本中添加 Scala-2.12.0 支持。

So you'll have to wait until they add support on future EMR releases or you may want to build a Spark with Scala 2.12 and install it on EMR.因此，您必须等到他们添加对未来 EMR 版本的支持，或者您可能想要使用 Scala 2.12 构建 Spark 并将其安装在 EMR 上。 See Building and Deploying Custom Applications with Apache Bigtop and Amazon EMR and Building a Spark Distribution for EMR .请参阅使用 Apache Bigtop 和 Amazon EMR 构建和部署自定义应用程序以及为 EMR 构建 Spark 分发。

UPDATE:更新：

Since Release 6.0.0 , Scala 2.12 can be used with Spark on EMR:从6.0.0 版开始，Scala 2.12 可以在 EMR 上与 Spark 一起使用：

Changes, Enhancements, and Resolved Issues更改、增强和已解决的问题

Scala斯卡拉
Scala 2.12 is used with Apache Spark and Apache Livy. Scala 2.12 与 Apache Spark 和 Apache Livy 一起使用。

Just an Idea, if waiting is not the option!只是一个想法，如果等待不是选择！

Is it possible to package the latest scala jars with the application with appropriate maven scope defined and point those packages with the spark property --properties spark.jars.repositories ??是否可以使用定义了适当 maven 范围的应用程序打包最新的 Scala jar，并使用 spark 属性--properties spark.jars.repositories指向这些包？

Maybe you'll have to figure out a way to transfer the jars to the driver node.也许您必须想办法将罐子传输到驱动程序节点。 If s3 is an option that can be used as intermediatory storage.如果 s3 是可用作中间存储的选项。