缩小时Grafana仪表板不显示数据

Question

I have a Prometheus server which scrapes our aws billing data as metrics using this exporter . I scrape the billing metrics every 6h to limit the requests. Than a Grafana server builds some dashboards using this data.

The dashboard shows data as expected when I select Last 24 hours

But if I change the range to this Month the Graph plots No data points

I have not found any way to get Grafana to show the data when zoomed out. This is the configuration

{
  "aliasColors": {},
  "bars": true,
  "dashLength": 10,
  "dashes": false,
  "fill": 1,
  "gridPos": { "h": 12, "w": 5, "x": 5, "y": 16 },
  "id": 12,
  "legend": {
    "alignAsTable": true,
    "avg": false,
    "current": true,
    "hideEmpty": false,
    "hideZero": false,
    "max": false,
    "min": false,
    "rightSide": false,
    "show": true,
    "total": false,
    "values": true
  },
  "lines": false,
  "linewidth": 1,
  "links": [],
  "nullPointMode": "connected",
  "percentage": false,
  "pointradius": 2,
  "points": false,
  "renderer": "flot",
  "seriesOverrides": [],
  "spaceLength": 10,
  "stack": true,
  "steppedLine": false,
  "targets": [
    {
      "expr": "topk(5 ,aws_billing_estimated_charges_sum{linked_account=\"1234567890\"})",
      "format": "time_series",
      "intervalFactor": 1,
      "legendFormat": "{{service_name}}",
      "refId": "A"
    }
  ],
  "thresholds": [],
  "timeFrom": null,
  "timeRegions": [],
  "timeShift": null,
  "title": "Prod: Top 5 Services ",
  "tooltip": {
    "shared": true,
    "sort": 0,
    "value_type": "individual"
  },
  "type": "graph",
  "xaxis": {
    "buckets": null,
    "mode": "time",
    "name": null,
    "show": true,
    "values": []
  },
  "yaxes": [
    {
      "format": "currencyUSD",
      "label": null,
      "logBase": 1,
      "max": null,
      "min": null,
      "show": true
    },
    {
      "format": "short",
      "label": null,
      "logBase": 1,
      "max": null,
      "min": null,
      "show": false
    }
  ],
  "yaxis": {
    "align": false,
    "alignLevel": null
  }
}

How can I get Grafana to show my chart?

Answer 1

I don't have any metrics with such low resolution, so I can't actually test this. But I suppose the problem is that the resulting step when zooming out is longer than 5 minutes (Prometheus' default "lookback delta", configurable via the --query.lookback-delta command-line flag).

What this lookback delta means is that if you query the value of a time series at a point in time (either via an instant query, or via a range query that will retrieve the values of said time series at distinct points in time, separated by a fixed step time) you'll only get back a value if there is a sample less than 5 minutes before. So with a step much larger than 5 minutes and samples much farther apart than 5 minutes, it's less likely for your query to hit timestamps less than 5 minutes after a sample and more likely to hit timestamps where there is no sample within the preceding 5 minutes.

So what you need to do is extend this 5 minute time range. Either via the command line flag above (not recommended, as it may break other assumptions, for time series with more reasonable scrape intervals); or by using one of the <aggregation>_over_time functions. In combination with Grafana's $__interval variable, something like this should work:

topk(5, max_over_time(aws_billing_estimated_charges_sum{linked_account="1234567890"}[$__interval]))

This would "force" Prometheus to look back $__interval (which is the interval between the samples Grafana queries for) rather than 5 minutes. With a time range of one month (and the graph size in your screenshot) I'd guess that would be somewhere around 1 hour, so Grafana would query for a sample every 1 hour, with the maximum value over the previous 1 hour.

If you want those samples to result in lines as in your first graph and not disjoint points, then you should also set a Min step of 6h in Grafana. Or use a fixed interval of 6h in your query, but then you'd get a step function that only changes every 6h .