简体繁体 English

从火花日志中提取指标

[英]pull out metrics from spark logs

原文 2022-12-06 17:17:37 2 1 apache-spark/ amazon-emr

how do I pull out these metrics from spark history logs?如何从 spark 历史日志中提取这些指标？ Is there some api I can pull these from?有一些 api 我可以从中提取这些吗？

I tried downloading the json event logs, but I can't grep for the numbers seen in the photo我尝试下载 json 事件日志，但我无法下载 grep 以获取照片中显示的数字

1 个解决方案

The spark history server keeps all that information for you. spark 历史服务器为您保留所有这些信息。 You can access it via a rest API.您可以通过 rest API 访问它。

If you are on EMR :如果您使用EMR ：

You can view the Spark web UIs by following the procedures to create an SSH tunnel or create a proxy in the section called Connect to the cluster in the Amazon EMR Management Guide and then navigating to the YARN ResourceManager for your cluster.您可以按照创建 SSH 隧道或创建代理的过程查看 Spark web UI，方法是在 Amazon EMR 管理指南中称为“连接到集群”的部分中创建代理，然后导航到集群的 YARN ResourceManager。 Choose the link under Tracking UI for your application.在您的应用程序的跟踪 UI 下选择链接。 If your application is running, you see ApplicationMaster.如果您的应用程序正在运行，您会看到 ApplicationMaster。 This takes you to the application master's web UI at port 20888 wherever the driver is located.这会将您带到应用程序主机的 web UI 端口 20888，无论驱动程序位于何处。 The driver may be located on the cluster's primary node if you run in YARN client mode.如果您在 YARN 客户端模式下运行，驱动程序可能位于集群的主节点上。 If you are running an application in YARN cluster mode, the driver is located in the ApplicationMaster for the application on the cluster.如果您在 YARN 集群模式下运行应用程序，则驱动程序位于集群上应用程序的 ApplicationMaster 中。 If your application has finished, you see History, which takes you to the Spark HistoryServer UI port number at 18080 of the EMR cluster's primary node.如果您的应用程序已完成，您会看到历史记录，它会将您带到 EMR 集群主节点的 18080 处的 Spark HistoryServer UI 端口号。 This is for applications that have already completed.这适用于已经完成的应用程序。 You can also navigate to the Spark HistoryServer UI directly at http://master-public-dns-name:18080/.您还可以通过 http://master-public-dns-name:18080/ 直接导航到 Spark HistoryServer UI。