[英]Spark AWS Glue UI - Error: Could not find or load main class $SPARK_HISTORY_OPTS - Windows Docker
Ref - https://github.com/aws-samples/aws-glue-samples/tree/master/utilities/Spark_UI/ https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-ui-history.html参考 - https://github.com/aws-samples/aws-glue-samples/tree/master/utilities/Spark_UI/ https://docs.aws.amazon.com/glue/latest/dg/monitor-spark- ui-history.html
I am trying to access glue spark ui locally on my windows 10 - (docker desktop) using docker container.我正在尝试使用 docker 容器在我的 windows 10 -(docker 桌面)上本地访问 glue spark ui。
I have followed following steps but I get error while starting container我已按照以下步骤操作,但在启动容器时出现错误
$ docker build -t glue/sparkui:latest .
$ LOG_DIR="s3a://path_to_eventlog/"
$ AWS_ACCESS_KEY_ID="ASIAxxxxxxxxxxxx"
$ AWS_SECRET_ACCESS_KEY="yyyyyyyyyyyyyyy"
$ SESSION_TOKEN="zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz"
$ docker run -itd -e SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=$LOG_DIR -Dspark.hadoop.fs.s3a.access.key=$AWS_ACCESS_KEY_ID -Dspark.hadoop.fs.s3a.secret.key=$AWS_SECRET_ACCESS_KEY -Dspark.hadoop.fs.s3a.session.token=$SESSION_TOKEN -Dspark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider" -p 18080:18080 glue/sparkui:latest "/opt/spark/bin/spark-class org.apache.spark.deploy.history.HistoryServer"
D:\glue_spark_gui_docker_file>docker logs 05c595fc8c65
Error: Could not find or load main class $SPARK_HISTORY_OPTS
same error when trying docker on wsl
with bash
on windows so I solved this using windows cmd
在 windows 上使用
bash
在 wsl 上尝试docker on wsl
时出现同样的错误,所以我使用 windows cmd
解决了这个问题
1. Build image 1.构建镜像
C:\Users\Avinash\Spark_UI\glue-1_0-2_0> docker build -t glue/sparkui:latest .
2. set variable 2.设置变量
set LOG_DIR="s3a://path_to_eventlog/"
set AWS_ACCESS_KEY_ID="XXXXYYYY"
set AWS_SECRET_ACCESS_KEY="SDFGHJHRFVBNMNBVC"
3. Run container remove $SPARK_HISTORY_OPTS
and add %variable%
instead of $variable
3. 运行 container remove
$SPARK_HISTORY_OPTS
并添加%variable%
而不是$variable
docker run -itd -e SPARK_HISTORY_OPTS="-Dspark.history.fs.logDirectory=%LOG_DIR% -Dspark.hadoop.fs.s3a.access.key=%AWS_ACCESS_KEY_ID% -Dspark.hadoop.fs.s3a.secret.key=%AWS_SECRET_ACCESS_KEY%" -p 18080:18080 --name sparkui glue/spark:latest "/opt/spark/bin/spark-class org.apache.spark.deploy.history.HistoryServer"
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.