[英]How can Apache Spark history-server refer to Amazon S3?
[version] [版]
Apache Spark 2.2.0 Apache Spark 2.2.0
Hadoop 2.7 Hadoop 2.7
I want to set up Apache Spark histroy server. 我想设置Apache Spark histroy服务器。
Spark events log located in Amazon S3. Spark事件日志位于Amazon S3中。
I can save log file in S3, but cannot read from history server. 我可以将日志文件保存在S3中,但不能从历史服务器中读取。
Apache Spark installed at /usr/local/spark
Apache Spark安装在
/usr/local/spark
so, $SPARK_HOME
is /usr/local/spark
因此,
$SPARK_HOME
是/usr/local/spark
$ cd /usr/local/spark/sbin
$ sh start-history-server.sh
I got following error 我收到以下错误
Exception in thread "main" java.lang.ClassNotFoundException: org.apache.hadoop.fs.s3a.S3AFileSystem
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:230)
....
my spark-defaults.conf
is below 我的
spark-defaults.conf
在下面
spark.hadoop.fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem
spark.history.provider org.apache.hadoop.fs.s3a.S3AFileSystem
spark.history.fs.logDirectory s3a://xxxxxxxxxxxxx
spark.eventLog.enabled true
spark.eventLog.dir s3a://xxxxxxxxxxxxxxx
I installed this 2 jar files in /usr/local/spark/jars/
我在
/usr/local/spark/jars/
安装了这2个jar文件
aws-java-sdk-1.7.4.jar
hadoop-aws-2.7.3.jar
but error is same. 但是错误是一样的。
What is wrong? 怎么了?
Please add the following in spark-defaults.conf file and retry again. 请在spark-defaults.conf文件中添加以下内容,然后重试。
spark.driver.extraClassPath :/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/* : spark.executor.extraClassPath :/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*: spark.driver.extraClassPath:/usr/lib/hadoop/hadoop-aws.jar:/ usr / share / aws / aws-java-sdk / *:spark.executor.extraClassPath:/ usr / lib / hadoop / hadoop-aws .jar:/ usr / share / aws / aws-java-sdk / *:
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.