[英]How to get all jobs status through spark REST API?
I am using spark 1.5.1 and I'd like to retrieve all jobs status through REST API.我正在使用 spark 1.5.1,我想通过 REST API 检索所有作业状态。
I am getting correct result using /api/v1/applications/{appId}
.我使用
/api/v1/applications/{appId}
得到正确的结果。 But while accessing jobs /api/v1/applications/{appId}/jobs
getting "no such app:{appID}" response.但是在访问作业时
/api/v1/applications/{appId}/jobs
得到“没有这样的应用程序:{appID}”响应。
How should I pass app ID here to retrieve jobs status of application using spark REST API?我应该如何在此处传递应用程序 ID 以使用 Spark REST API 检索应用程序的作业状态?
Spark provides 4 hidden RESTFUL API Spark 提供 4 个隐藏的 RESTFUL API
1) Submit the job - curl -X POST http://SPARK_MASTER_IP:6066/v1/submissions/create 1) 提交作业 - curl -X POST http://SPARK_MASTER_IP:6066/v1/submissions/create
2) To kill the job - curl -X POST http://SPARK_MASTER_IP:6066/v1/submissions/kill/driver-id 2)终止工作 - curl -X POST http://SPARK_MASTER_IP:6066/v1/submissions/kill/driver-id
3) To check status if the job - curl http://SPARK_MASTER_IP:6066/v1/submissions/status/driver-id 3)检查工作状态 - curl http://SPARK_MASTER_IP:6066/v1/submissions/status/driver-id
4) Status of the Spark Cluster - http://SPARK_MASTER_IP:8080/json/ 4) Spark 集群状态 - http://SPARK_MASTER_IP:8080/json/
If you want to use another APIs you can try Livy , lucidworks url - https://doc.lucidworks.com/fusion/3.0/Spark_ML/Spark-Getting-Started.html如果您想使用其他 API,您可以尝试 Livy ,lucidworks url - https://doc.lucidworks.com/fusion/3.0/Spark_ML/Spark-Getting-Started.html
This is supposed to work when accessing a live driver's API endpoints, but since you're using Spark 1.5.x I think you're running into SPARK-10531 , a bug where the Spark Driver UI incorrectly mixes up application names and application ids.这在访问实时驱动程序的 API 端点时应该可以工作,但由于您使用的是 Spark 1.5.x,我认为您遇到了SPARK-10531 ,这是一个错误,即 Spark 驱动程序 UI 错误地混淆了应用程序名称和应用程序 ID。 As a result, you have to use the application name in the REST API url, eg
因此,您必须在 REST API url 中使用应用程序名称,例如
http://localhost:4040/api/v1/applications/Spark%20shell/jobs
According to the JIRA ticket, this only affects the Spark Driver UI;根据 JIRA 票证,这仅影响 Spark Driver UI; application IDs should work as expected with the Spark History Server's API endpoints.
应用程序 ID 应该与 Spark History Server 的 API 端点一起按预期工作。
This is fixed in Spark 1.6.0, which should be released soon.这在 Spark 1.6.0 中得到了修复,它应该很快就会发布。 If you want a workaround which should work on all Spark versions, though, then the following approach should work:
但是,如果您想要一种适用于所有 Spark 版本的解决方法,那么以下方法应该适用:
The api/v1/applications
endpoint misreports job names as job ids, so you should be able to hit that endpoint, extract the id
field (which is actually an application name), then use that to construct the URL for the current application's job list (note that the /applications
endpoint will only ever return a single job in the Spark Driver UI, which is why this approach should be safe; due to this property, we don't have to worry about the non-uniqueness of application names). api/v1/applications
端点将作业名称误报为作业 ID,因此您应该能够访问该端点,提取id
字段(实际上是应用程序名称),然后使用它来构建当前应用程序作业列表的 URL (注意/applications
端点只会在 Spark Driver UI 中返回一个作业,这就是为什么这种方法应该是安全的;由于这个属性,我们不必担心应用程序名称的非唯一性) . For example, in Spark 1.5.2 the /applications
endpoint can return a response which contains a record like例如,在 Spark 1.5.2 中,
/applications
端点可以返回一个包含类似记录的响应
{
id: "Spark shell",
name: "Spark shell",
attempts: [
{
startTime: "2015-09-10T06:38:21.528GMT",
endTime: "1969-12-31T23:59:59.999GMT",
sparkUser: "",
completed: false
}]
}
If you use the contents of this id
field to construct the applications/<id>/jobs
URL then your code should be future-proofed against upgrades to Spark 1.6.0, since the id
field will begin reporting the proper IDs in Spark 1.6.0+.如果您使用此
id
字段的内容来构建applications/<id>/jobs
URL,那么您的代码应该能够适应未来升级到 Spark 1.6.0,因为id
字段将开始报告 Spark 1.6 中的正确 ID。 0+。
If you want to use the REST API to control Spark, you're probably best adding the Spark Jobserver to your installation which then gives you a much more comprehensive REST API than the private REST APIs you're currently querying.如果您想使用 REST API 来控制 Spark,您可能最好将Spark 作业服务器添加到您的安装中,然后它会为您提供比您当前查询的私有 REST API 更全面的 REST API。
Poking around, I've managed to get the job status for a single application by running四处闲逛,我设法通过运行获得了单个应用程序的作业状态
curl http://127.0.0.1:4040/api/v1/applications/Spark%20shell/jobs/
curl http://127.0.0.1:4040/api/v1/applications/Spark%20shell/jobs/
which returned返回
[ {
"jobId" : 0,
"name" : "parquet at <console>:19",
"submissionTime" : "2015-12-21T10:46:02.682GMT",
"stageIds" : [ 0 ],
"status" : "RUNNING",
"numTasks" : 2,
"numActiveTasks" : 2,
"numCompletedTasks" : 0,
"numSkippedTasks" : 0,
"numFailedTasks" : 0,
"numActiveStages" : 1,
"numCompletedStages" : 0,
"numSkippedStages" : 0,
"numFailedStages" : 0 }]
For those who have this problem and are running on YARN:对于那些有这个问题并在 YARN 上运行的人:
when running in YARN cluster mode, [app-id] will actually be [base-app-id]/[attempt-id], where [base-app-id] is the YARN application ID
在 YARN 集群模式下运行时,[app-id] 实际上是 [base-app-id]/[attempt-id],其中 [base-app-id] 是 YARN 应用程序 ID
So if your call to https://HOST:PORT/api/v1/applications/application_12345678_0123
returns something like因此,如果您对
https://HOST:PORT/api/v1/applications/application_12345678_0123
的调用返回类似
{
"id" : "application_12345678_0123",
"name" : "some_name",
"attempts" : [ {
"attemptId" : "1",
<...snip...>
} ]
}
you can get eg.你可以得到例如。 jobs by calling
打电话找工作
https://HOST:PORT/api/v1/applications/application_12345678_0123/1/jobs
(note the "1" before "/jobs"). (注意“/jobs”之前的“1”)。
Spark has some hidden RESTFUL API that you can try. Spark 有一些隐藏的 RESTFUL API,你可以试试。 Note that i have not tried yet, but i will.
请注意,我还没有尝试过,但我会尝试的。
For example: to get status of submit application you can do: curl http://spark-cluster-ip:6066/v1/submissions/status/driver-20151008145126-0000例如:要获取提交申请的状态,您可以执行以下操作:curl http://spark-cluster-ip:6066/v1/submissions/status/driver-20151008145126-0000
Note: "driver-20151008145126-0000" is submitsionId.注意:“driver-20151008145126-0000”是submitsionId。
You can take a deep look in this link with this post from arturmkrtchyan on GitHub您可以通过 GitHub 上 arturmkrtchyan 的这篇文章深入了解此链接
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.