How to view AWS Glue Spark UI

Question

In my Glue job, I have enabled Spark UI and specified all the necessary details (s3 related etc.) needed for Spark UI to work.
How can I view the DAG/Spark UI of my Glue job?

Answer 1

You need to setup an ec2 instance that can host the history server.

The below documentation has links to CloudFormation templates that you can use. https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-ui-history.html

You can access the history server via the ec2 instance(default on 18080). You need to configure the networks and ports suitably.

EDIT - There is also an option to setup SparkUI locally. This requires downloading the docker image from aws-glue-samples repo amd settin the AWS credential and s3 location there. This server consummes the files that the glue job generates. The files are about 4MB large.