Grafana Prometheus Counter

Question

I have trying to get exact count for an event in a Grafana visualization using Prometheus as timeseries DB. But the counter is showing incorrect records. I am getting a higher count for 2 days that I am getting 7 days, which definately points to something being wrong.

First I have used a single stats visualization with this promql query:

sum(increase(http_server_requests_seconds_count[$__range])).

PS

I have also tried the following: sum(increase(http_server_requests_seconds_count[1m])) . This also gives incorrect counts.

I have tried the same with graph and using the legend to show totals in table. This is also gives incorrect counts.

Please let me know what is the best way of showing counts which can be reliable over time range changes.

My json:

{
    "colorMode": "value",
    "fieldOptions": {
        "calcs": [
            "lastNotNull"
        ],
        "defaults": {
            "mappings": [],
            "thresholds": {
                "mode": "absolute",
                "steps": [{
                    "color": "green",
                    "value": null
                }]
            }
        },
        "overrides": [],
        "values": false
    },
    "graphMode": "area",
    "justifyMode": "auto",
    "orientation": "auto"
},
"pluginVersion": "6.6.1",
"targets": [{
    "expr": " sum(increase(http_server_requests_seconds_count[$__range]))",
    "hide": false,
    "instant": true,
    "refId": "A"
}],
"timeFrom": null,
"timeShift": null,
"title": "Total Number of Requests",
"type": "stat"
}

Answer 1

This works for me:

sum(increase(http_request_duration_seconds_count{ecs_cluster=~"$ecs_cluster", instance_id=~"$instance_id"}[$__range]))

Activated instant query and set calculation to last not null

Here is the pane JSON:

{
  "cacheTimeout": null,
  "datasource": "Prometheus",
  "description": "",
  "fieldConfig": {
    "defaults": {
      "custom": {},
      "unit": " requests",
      "decimals": 0,
      "thresholds": {
        "mode": "absolute",
        "steps": [
          {
            "color": "blue",
            "value": null
          }
        ]
      },
      "mappings": [],
      "nullValueMode": "connected"
    },
    "overrides": []
  },
  "gridPos": {
    "h": 2,
    "w": 5,
    "x": 0,
    "y": 4
  },
  "id": 4,
  "interval": null,
  "links": [],
  "maxDataPoints": 100,
  "options": {
    "reduceOptions": {
      "values": false,
      "calcs": [
        "lastNotNull"
      ],
      "fields": ""
    },
    "orientation": "horizontal",
    "textMode": "auto",
    "colorMode": "value",
    "graphMode": "none",
    "justifyMode": "auto",
    "fieldOptions": {
      "calcs": [
        "lastNotNull"
      ]
    }
  },
  "pluginVersion": "7.1.0",
  "targets": [
    {
      "expr": "sum(increase(http_request_duration_seconds_count{ecs_cluster=~\"$ecs_cluster\", instance_id=~\"$instance_id\"}[$__range]))",
      "hide": false,
      "instant": true,
      "interval": "",
      "intervalFactor": 1,
      "legendFormat": "",
      "refId": "A"
    }
  ],
  "timeFrom": null,
  "timeShift": null,
  "title": "",
  "type": "stat"
}

Answer 2

Prometheus may return inaccurate results from increase() function because of the chosen data model - see this issue for details.

If you need accurate results, then the following options exist:

• To use offset . Try something like the following: sum(http_server_requests_seconds_count - http_server_requests_seconds_count offset $__range) . Note that this approach works only if the given metric - http_server_requests_seconds_count wasn't reset to 0 (aka counter reset ) on the given time range.
• To use increase() function from MetricsQL. It returns accurate values - see these docs for details.