Get console job output text from dataproc using rest api

Question

I need to retrieve the dataproc job output text using the rest api. Only able to find logs through cloud logging. Can someone let me know if it is possible to get the job output text retrieved through rest api or not. If yes how?

Answer 1

Dataproc job driver output is stored in either the staging bucket (default) or the bucket you specified when you created your cluster. You can first get the job resource, then get the URI through driverOutputResourceUri , then use GCS API to get the actual output. See this doc for more details.

If you use gcloud you can get driverOutputResourceUri with describe :

$ gcloud dataproc jobs describe spark-pi

...
driverOutputResourceUri: gs://dataproc-nnn/jobs/spark-pi/driveroutput

If you just want to view the output, you can use wait :

$ gcloud dataproc jobs wait 5c1754a5-34f7-4553-b667-8a1199cb9cab \
    --project my-project-id --region my-cluster-region

Waiting for job output...
INFO gcs.GoogleHadoopFileSystemBase: GHFS version: 1.4.2-hadoop2 16:47:45 INFO client.RMProxy: Connecting to ResourceManager at my-test-cluster-m/
...