Grafana settings lost when host ethernet interface goes down and back up

Reporters

• Mika Silvola (Infinera)
• Lluis Gifre (CTTC)

Description

Host running TFS had network issues and interface went down and back up. As results Grafana restarted and comes back on default settings without TFS related templates. To recovery from situation TFS must be restarted. Root cause of the problem is that deploy/tfs.sh scripts configures Granana setup via script once, but this settings is not re-played in case of Grafana restart.

Deployment environment

• Ubunut 22.04
• MicroK8s (1.2.24):

  enabled:
    linkerd              # (community) Linkerd is a service mesh for Kubernetes and other frameworks
    community            # (core) The community addons repository
    dns                  # (core) CoreDNS
    ha-cluster           # (core) Configure high availability on the current node
    helm3                # (core) Helm 3 - Kubernetes package manager
    hostpath-storage     # (core) Storage class; allocates storage from host directory
    ingress              # (core) Ingress controller for external access
    metrics-server       # (core) K8s Metrics Server for API access to service metrics
    prometheus           # (core) Prometheus operator for monitoring and logging
    registry             # (core) Private image registry exposed on localhost:32000
    storage              # (core) Alias to hostpath-storage add-on, deprecated

• TeraFlowSDN (include release/branch-name/commit-id): commit 7335ba27 (origin/develop) Date: Thu Feb 8 10:35:40 2024 +0000

TFS deployment settings

• base/minimal settings, +ztp and +monitoring enabled

Sequence of actions that resulted in the bug

• easy to re-produce:

• create setup which generates statistic events and monitor via grafana

• sudo ifconfig down

• grafana restarts and after comes back has no TFS settings present

• sudo ifconfig up

• Lluis gave quick analysis:

  • probably the reason is because the failure causes grafana to self-redeploy and grafana settings are not persisted in disk (it uses a deployment set not a stateful set)

Document the explicit error

• grafana lost settings

Expected behaviour

• Expected behavior would be that Grafana doesn't restart on interface down/up transition, or alternatively in case restart re-plays TFS related settings back.

References