在 Prometheus 警报中的活动 state 中无法看到警报

Question

I am trying to setup alert manager with my prometheus server. i can see following alert is triggered properly and notification is available in my slack channel. it is with simple expression and triggering if any exporter is down

ie up== 0

alerts:
    groups:
      - name: Exporter
        rules:
        - alert: exporter-down
          expr: up == 0
          for: 2m
          labels:
            severity: page
          annotations:
            Summary: "Exporter {{ $labels.job }} is down."
            Description: "{{ $labels.job }} has been down for more than 2 minutes."
            GrafanaDashboard: example.com
            Prometheus: example.com
            AlertManager: example.com
            Impact: Unavailability of {{ $labels.job }} will impact our monitoring. We will not able to get Insight of {{ $labels.job }}

when i try to set similar alert rule for response in some specific range.

ie expr aws_applicationelb_target_response_time_average > 0.1 AND aws_applicationelb_target_response_time_average < 0.35

- name: LoadBalancerWarning
        rules:
        - alert: slowResponseWarning
          expr: aws_applicationelb_target_response_time_average > 0.1 AND aws_applicationelb_target_response_time_average < 0.35
          labels:
            severity: warning
          annotations:
            Summary: "Load Balancer {{ $labels.load_balancer }} response is more than 0.1 seconds but less than 0.35 for {{ $labels.job }} "
            Description: "It is Warning Sign. "
            GrafanaDashboard: example.com
            Prometheus: example.com
            AlertManager: example.com
            Impact: Slow Response Impact User Experience  

Not sure if i am missing something. it will be great if someone can provide me pointers for same.

Answer 1

I can see cloud watch has delay of 5 to 10 minutes. when i add offset in this alert query for 5 to 10 minutes. it is able to trigger alerts