pull out metrics from spark logs

Question

how do I pull out these metrics from spark history logs? Is there some api I can pull these from?

I tried downloading the json event logs, but I can't grep for the numbers seen in the photo

Answer 1

The spark history server keeps all that information for you. You can access it via a rest API.

If you are on EMR :

You can view the Spark web UIs by following the procedures to create an SSH tunnel or create a proxy in the section called Connect to the cluster in the Amazon EMR Management Guide and then navigating to the YARN ResourceManager for your cluster. Choose the link under Tracking UI for your application. If your application is running, you see ApplicationMaster. This takes you to the application master's web UI at port 20888 wherever the driver is located. The driver may be located on the cluster's primary node if you run in YARN client mode. If you are running an application in YARN cluster mode, the driver is located in the ApplicationMaster for the application on the cluster. If your application has finished, you see History, which takes you to the Spark HistoryServer UI port number at 18080 of the EMR cluster's primary node. This is for applications that have already completed. You can also navigate to the Spark HistoryServer UI directly at http://master-public-dns-name:18080/.