fix: instruct an alert rule to maintain its previous state when a datasource disconnection occurs

Labros Papadopoulosrequested to merge
fix-ghost-alerts-datasource-error into main

In Grafana, the execErrState: KeepLastState setting instructs an alert rule to maintain its previous state (e.g., Normal, Alerting, or Pending) when the underlying query execution fails or produces an unexpected error. It prevents false alarms caused by temporary network timeouts or database disconnects. In our case when core dns fails and Grafana loses connectivity with Loki datasource it will prevent it from creating a false alarm.

Merge request reports

charts/hypo/templates/config/fabric-api-rules.yaml

+1 −1
Changes for charts/hypo/templates/config/fabric-api-rules.yaml: 1 added line, 1 removed line.
Original line number Diff line number Diff line
@@ -68,7 +68,7 @@ data: 
                      type: query

                  refId: C

            noDataState: OK

            execErrState: Error

            execErrState: KeepLastState

            for: 0s

            annotations:

              summary: 'Fabric API Error in {{ "{{" }} $labels.container {{ "}}" }}'

charts/hypo/templates/config/oss-client-rules.yaml

+1 −1
Changes for charts/hypo/templates/config/oss-client-rules.yaml: 1 added line, 1 removed line.
Original line number Diff line number Diff line
@@ -68,7 +68,7 @@ data: 
                      type: query

                  refId: C

            noDataState: OK

            execErrState: Error

            execErrState: KeepLastState

            for: 0s

            annotations:

              summary: 'Oss client Error in {{ "{{" }} $labels.container {{ "}}" }}'

charts/hypo/templates/config/peering-api-rules.yaml

+1 −1
Changes for charts/hypo/templates/config/peering-api-rules.yaml: 1 added line, 1 removed line.
Original line number Diff line number Diff line
@@ -70,7 +70,7 @@ data: 
                      type: query

                  refId: C

            noDataState: OK

            execErrState: Error

            execErrState: KeepLastState

            for: 0s

            annotations:

              summary: 'Peering Error in {{ "{{" }} $labels.container {{ "}}" }}'

charts/hypo/templates/config/pkg-manager-rules.yaml

+1 −1
Changes for charts/hypo/templates/config/pkg-manager-rules.yaml: 1 added line, 1 removed line.
Original line number Diff line number Diff line
@@ -68,7 +68,7 @@ data: 
                      type: query

                  refId: C

            noDataState: OK

            execErrState: Error

            execErrState: KeepLastState

            for: 0s

            annotations:

              summary: 'Pkg Manager Error in {{ "{{" }} $labels.container {{ "}}" }}'

charts/hypo/templates/config/registry-api-rules.yaml

+1 −1
Changes for charts/hypo/templates/config/registry-api-rules.yaml: 1 added line, 1 removed line.
Original line number Diff line number Diff line
@@ -68,7 +68,7 @@ data: 
                      type: query

                  refId: C

            noDataState: OK

            execErrState: Error

            execErrState: KeepLastState

            for: 0s

            annotations:

              summary: 'Registry API Error in {{ "{{" }} $labels.container {{ "}}" }}'