Programmatically determine number of cores and amount of memory available to Spark

Question

The Spark Web UI shows some interesting information about the resources available to the cluster as a whole.

I'm specifically interested in the values for:

• Workers
• Cores
• Memory

How can I query these pieces of information about the overall cluster programmatically?

Answer 1

Spark doesn't really expose this kind of info, it's all hidden in the Master and transferred to the WebUI.

You can however use a small hack, the WebUI supports JSON by appending /json/ to a page.

So, going to http://<master-host>:<master-port>/json/ will return just the info you're looking for:

{
  url: "spark://<host>:<port>",
  workers: [ ],
  cores: 0,
  coresused: 0,
  memory: 0,
  memoryused: 0,
  activeapps: [ ],
  completedapps: [ ],
  activedrivers: [ ],
  status: "ALIVE"
}