[英]How to save and use Spark History Server logs in AWS S3
我想在 AWS S3 中記錄和查看 Spark History Server 的事件日志。
以下是 spark-defaults.conf 中記錄的屬性。
spark.hadoop.fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.endpoint {{endpoint}}
spark.hadoop.fs.s3a.access.key {{accessKey}}
spark.hadoop.fs.s3a.secret.key {{secretKey}}
spark.hadoop.fs.s3a.fast.upload true
spark.hadoop.fs.s3a.block.size 268435456
spark.eventLog.enabled true
spark.eventLog.dir s3a://{{bucketName}}/{{path}}
spark.history.fs.logDirectory s3a://{{bucketName}}/{{path}}
但是在啟動spark歷史服務器時,出現如下錯誤。
20/10/07 14:07:14 INFO S3AFileSystem: Error Message: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: {{requestId}}, AWS Error Code: null, AWS Error Message: Forbidden
20/10/07 14:07:14 INFO S3AFileSystem: HTTP Status Code: 403
20/10/07 14:07:14 INFO S3AFileSystem: AWS Error Code: null
20/10/07 14:07:14 INFO S3AFileSystem: Error Type: Client
20/10/07 14:07:14 INFO S3AFileSystem: Request ID: {{requestId}}
20/10/07 14:07:14 INFO S3AFileSystem: Class Name: com.amazonaws.services.s3.model.AmazonS3Exception
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:296)
at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: {{requestId}}, AWS Error Code: null, AWS Error Message: Forbidden, S3 Extended Request ID: {{requestId}}
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:976)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:956)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:688)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:71)
at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:257)
at org.apache.spark.deploy.history.FsHistoryProvider.initialize(FsHistoryProvider.scala:211)
at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:207)
at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:86)
... 6 more
使用上述認證信息調用API時,上傳和下載都很好。
配置 Spark History Server 時我錯過了什么嗎?
我能夠讓這個工作。 基本上你需要將相關的 jars 添加到 Spark_Home/jars 目錄中。 請找到我對類似問題的詳細答案: https : //stackoverflow.com/a/65086818/6239561
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.