Metrics not showing on Grafana dashboard, at some points

Question

I have set up a Grafana dashboard, to monitor Kubernetes. It's working as expected but there's an issue of losing metrics/history at some point of time. See the attached screenshot. In the screenshot, no metrics are shown between 21:00 PM to 23:00 PM.

https://imgur.com/lQFKV2W

From the above image, is that behavior normal? Thanks.

I'm using Grafana as an addon on Istio.

Part of the Istio helm chart configuration:

grafana:
  enabled: true
  persist: true  
  accessMode: ReadWriteMany

My expectation is that no history/metrics should be lost at any specific time.

Answer 1

Istio uses Prometheus to collect the metrics. What you want to do, is to change a Prometheus data retention.

For example, if you are installing it with Helm, just change this parameter in the chart

--set prometheus.retention=2d

source: https://github.com/istio/istio/issues/12567