基于变量的多个主机的 Grafana Graph CPU

Question

I am using GrafanaCloud. I have a dashboard variable to create a list of nodes:

node -> label_values(agent_hostname)

Then I have my query to graph CPU for the selected hostname:

instance:node_cpu_utilisation:rate5m{agent_hostname="$node"}

This works fine for a single host but I would like to be able to have lines for several servers on one graph. I have 'Include All' and 'Multi-value' switched on for the variable. At the moment, when I choose a second or third server from my nodes variable the graph shows 'No data'. Do I need to amend my variable so that it parses with a pipe (OR) symbol at the end? And if so how would I do that?

Answer 1

I still haven't solved this but I managed the following workaround. I added the following to the grafana-agent.yaml file so that prometheus would add an environment and role label for each machine:

  prometheus_remote_write:
    url: {{ prometheus_url }}
    write_relabel_configs:
      - source_labels: [__address__]
        regex: '.*'
        target_label: instance
        replacement: {{ ansible_hostname }}
      - source_labels: [__address__]
        regex: '.*'
        target_label: environment
        replacement: {{ env | default('legacy') }}
      - source_labels: [__address__]
        regex: '.*'
        target_label: role
        replacement: {{ role | default('none') }}

Then in the dashboard variables I added:

Then in the query for each metric I added the role and environment, for example for load:

node_load15{role="$role", environment="$environment" }

This allows me to show load for several machines on the one graph and also allows me to easily switch between environments and clusters using the variables drop-down at the top.