[英]How to get the status of a Step in AWS EMR cluster?
我正在嘗試創建一個EMR集群並執行一個spark Job。
我需要啟動一個線程,這樣當我的工作完成時,讓我知道該步驟已經完成,因為我的EMR集群將啟動並運行。
{
AmazonElasticMapReduceClient emr = configureEMRClient();
StepFactory stepFactory = new StepFactory();
StepConfig enableDebugging = new StepConfig()
.withName("Enable Debugging")
.withActionOnFailure("TERMINATE_JOB_FLOW")
.withHadoopJarStep(stepFactory.newEnableDebuggingStep());
HadoopJarStepConfig runExampleConfig = new HadoopJarStepConfig()
.withJar("command-runner.jar")
.withArgs("spark-submit")
.withArgs("--deploy-mode","cluster")
.withArgs("--master","yarn")
.withArgs("--class", extractorMainClass )
.withArgs("--num-executors","3")
.withArgs("--driver-memory","8g")
.withArgs( resourceExtractorJar )
.withArgs("st")
.withArgs("ap");
StepConfig customExampleStep = new StepConfig()
.withName("Example Step")
.withActionOnFailure("TERMINATE_JOB_FLOW")
.withHadoopJarStep(runExampleConfig);
Application spark = new Application().withName("Spark");
RunJobFlowRequest request = new RunJobFlowRequest()
.withName("Test")
.withReleaseLabel("emr-5.5.0")
.withSteps(enableDebugging, customExampleStep)
.withApplications(spark)
.withServiceRole("EMR_DefaultRole")
.withJobFlowRole("EMR_EC2_DefaultRole")
.withInstances(new JobFlowInstancesConfig()
.withEc2KeyName("keypair")
.withInstanceCount(2)
.withKeepJobFlowAliveWhenNoSteps(false)
.withMasterInstanceType("m3.xlarge")
.withSlaveInstanceType("m3.xlarge"));
RunJobFlowResult result = emr.runJobFlow(request);
}
要獲得步驟狀態:
while(true) {
DescribeClusterRequest desc = new DescribeClusterRequest()
.withClusterId(jobFlowId);
DescribeClusterResult clusterResult = emrClient.describeCluster(desc);
com.amazonaws.services.elasticmapreduce.model.Cluster cluster = clusterResult.getCluster();
DescribeStepRequest d = new DescribeStepRequest().withClusterId(jobFlowId);
DescribeStepResult r = emrClient.describeStep(d);
Step ss = r.getStep();
String status = ss.getStatus().getState();
System.out.printf("Status: %s\n", status);
if(status.equals(ClusterState.TERMINATED.toString()) || status.equals(ClusterState.TERMINATED_WITH_ERRORS.toString())) {
System.out.println("Terminated");
break;
}
try {
TimeUnit.SECONDS.sleep(30);
} catch (InterruptedException e) {
e.printStackTrace();
}
// maybe other handle
}
收到以下錯誤:
Exception in thread "main" com.amazonaws.services.elasticmapreduce.model.InvalidRequestException: Step id 'null' is not valid. (Service: AmazonElasticMapReduce; Status Code: 400; Error Code: InvalidRequestException; Request ID: 98e1b43e-a440-11e7-920b-158124595c35)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
如何獲取EMR集群部署中的步驟狀態?
回答這個問題,因為我發現:
ListStepsResult steps = emrClient.listSteps(new ListStepsRequest().withClusterId(jobFlowId));
StepSummary step = steps.getSteps().get(0);
System.out.println(step.getStatus().getState());
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.