[英]AWS EMR cluster with Flink does not run any Jar, instead gives java.lang.NoSuchMethodError
I'm trying to submit a jar as a step in AWS EMR using the following command:我正在尝试使用以下命令提交 jar 作为 AWS EMR 中的一个步骤:
aws emr add-steps --cluster-id j-XXXXXXXXX --steps Type=CUSTOM_JAR,Name=test-job,ActionOnFailure=CONTINUE,Jar=s3://xxxx/frauddetection-0.1.jar,Args=flink-yarn-session,-d
It is a basic EMR cluster with Flink 1.14.2 and JDK 8 Adding this as a step gave the error:它是一个带有 Flink 1.14.2 和 JDK 8 的基本 EMR 集群 添加这个作为一个步骤给出了错误:
java.lang.NoSuchMethodError:org.apache.commons.math3.stat.descriptive.rank.Percentile.withNaNStrategy
java.lang.NoSuchMethodError:org.apache.commons.math3.stat.descriptive.rank.Percentile.withNaNStrategy
I checked the Jar and apache commoms library is present in it.我检查了其中存在 Jar 和 apache comcoms 库。 JDK and Flink version on my local machine are the same as that on EMR and the jar runs without any issue locally
我本地机器上的 JDK 和 Flink 版本与 EMR 上的相同,jar 在本地运行没有任何问题
One possibility is that a different (typically older) version of the Apache Commons Math3 jar is part of the AWS EMR installation, and so that's what is getting picked up first on the classpath.一种可能性是 Apache Commons Math3 jar 的不同(通常较旧)版本是 AWS EMR 安装的一部分,因此这就是在类路径上首先被拾取的。
You could try shading those classes when you build your jar.您可以在构建 jar 时尝试对这些类进行着色。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.