Commit 69bcae8b authored by Kevin Di Lallo's avatar Kevin Di Lallo
Browse files

upgrade to support long-term storage

parent 2c48b55b

README.md

+81 −10
Original line number Diff line number Diff line
@@ -35,7 +35,11 @@ To install the MEC Sandbox, follow the installation instructions below. 
### Prerequisites

MEC Sandbox has the following prerequisites:



1. [AdvantEDGE](https://github.com/InterDigitalInc/AdvantEDGE/wiki) runtime & development environments

1. MEC Sandbox (AdvantEDGE + MEC Sandbox frontend) VM:

   - [Hardware requirements](https://github.com/InterDigitalInc/AdvantEDGE/wiki/env-hw)

1. Long-term Storage (MinIO + Thanos Compactor) VM:

   - [Hardware requirements](https://github.com/InterDigitalInc/AdvantEDGE/wiki/env-hw)

1. [AdvantEDGE](https://github.com/InterDigitalInc/AdvantEDGE/wiki) runtime & development environments on both VMs

   - Install [runtime environment](https://github.com/InterDigitalInc/AdvantEDGE/wiki/runtime-environment)

   - Install [development environment](https://github.com/InterDigitalInc/AdvantEDGE/wiki/development-environment)

1. OAuth Applications using external providers
@@ -58,8 +62,62 @@ MEC Sandbox has the following prerequisites: 
     - **NOTES:**

       - Once created, keep note of the OAuth App Client ID & Secret



### Build & Deploy

Build and deploy the MEC Sandbox as follows:

### Deploy Long-term Storage (MinIO + Thanos Compactor)



1. Clone AdvantEDGE repository:

   - Fetch required AdvantEDGE baseline:

     - Clone AdvantEDGE repo to the Long-term Storage VM

     - Place repo in home folder

     - Checkout the required AdvantEDGE version tag

1. Configure Long-term Storage:

   - Configure object store:

     - `vi ~/AdvantEDGE/config/secrets.yaml`

       - Set minio accessKey to object store access key value (minimum 3 chars)

       - Set minio secretKey to object store secret key value (minimum 8 chars)

     - `vi ~/AdvantEDGE/config/thanos.yaml`

       - Set endpoint to VM FQDN

         - **NOTE:** Make sure _/etc/hosts_ file contains entry for VM FQDN that points to VM private IP address

       - Set access_key to object store access key value 

       - Set secret_key to object store secret key value

   - Configure deployment:

     - `vi ~/AdvantEDGE/config/long-term-storage-repocfg.yaml`

       - Set uid & gid

       - Set host name to FQDN

       - Set Thanos compactor retention times

     - Overwrite repocfg file with provided long-term storage repocfg:

       - `cp ~/AdvantEDGE/config/long-term-storage-repocfg.yaml ~/AdvantEDGE/.meepctl-repocfg.yaml`

1. Create MinIO & Thanos Secrets:

   - MinIO:

     - Install Python2:

       - Ubuntu 18.04: `sudo apt install python-minimal`

       - Ubuntu 20.04: `sudo apt install python2-minimal`

     - Install Python-pip:

       - Ubuntu 18.04: `sudo apt install python-pip`

       - Ubuntu 20.04: `sudo apt install python3-pip`

     - Install Pyyaml: `pip install pyyaml`

     - Create secrets:

       - Ubuntu 18.04: `python ~/AdvantEDGE/config/configure-secrets.py set ~/AdvantEDGE/config/secrets.yaml`

       - Ubuntu 20.04: `python3 ~/AdvantEDGE/config/configure-secrets.py set ~/AdvantEDGE/config/secrets.yaml`

   - Thanos:

     - `~/AdvantEDGE/config/configure-thanos-secrets.sh`

   - **NOTE:** Once the kubernetes secrets have been created, it is recommended to remove the config files containing the clear-text secrets

1. Deploy MinIO & Thanos Compactor:

   - Build meepctl:

     - `~/AdvantEDGE/go-apps/meepctl/install.sh`

     - **NOTE:** On first install the IP address & gitdir must be configured

   - Deploy dependency pods:

     - `meepctl deploy dep`

   - Deploy Core pods:

     - `meepctl deploy core`

     - **NOTE:** Core pods deployment will only install the cert-manager rules to trigger certificate installation

1. Log In & Create buckets:

   - Using a browser, access the long-term storage VM via its FQDN

   - Sign in using the provisioned credentials (access key & secret key)

   - Create the following buckets if they do not exist:

     - thanos

     - thanos-archive



### Build & Deploy MEC Sandbox (AdvantEDGE + MEC Sandbox Frontend)



1. Stop AdvantEDGE (if running)

   - Delete user sandboxes
@@ -82,10 +140,20 @@ Build and deploy the MEC Sandbox as follows: 
     - Place repo in home folder

     - Checkout the required AdvantEDGE version tag

1. Configure MEC Sandbox:

   - Configure object store:

     - `vi ~/AdvantEDGE/config/thanos.yaml` and `vi ~/AdvantEDGE/config/thanos-archive.yaml`

       - Set endpoint to long-term storage VM FQDN

         - **NOTE:** Make sure _/etc/hosts_ file contains entry for VM FQDN that points to VM private IP address (if on same local network)

       - Set access_key to object store access key value 

       - Set secret_key to object store secret key value

   - Configure session & OAuth secrets:

     - `vi ~/mec-sandbox/config/secrets.yaml`

       - Set a unique session encryption key

       - Set the OAuth provider client ID & secrets using the values from the OAuth Applications

   - Configure deployment:

     - `vi ~/mec-sandbox/config/.meepctl-repocfg.yaml`

       - Set Prometheus external labels

         - **NOTE:** This uniquely identifies Prometheus data sources in the long-term storage data buckets

   - Configure Alert Manager slack channel webhook:

     - Create incoming webhook:

       - Create target slack channel in slack workspace
@@ -110,13 +178,16 @@ Build and deploy the MEC Sandbox as follows: 
     - `./deploy.sh`

     - **NOTE:** This updates the `~/.meep/user/` folder with the user frontend, sandbox swagger UI, and user chart values.yaml overrides

     - **NOTE:** This also copies the AdvantEDGE configuration files to their respective locations in your local AdvantEDGE clone `~/AdvantEDGE/`. Running `git status` in your AdvantEDGE clone will list the updated (or new) files.

1. Create Session & OAuth Secrets:

1. Create Secrets:

   - Session & OAuth:

     - Install Python2: `sudo apt install python-minimal`

     - Install Python-pip: `sudo apt install python-pip`

     - Install Pyyaml: `pip install pyyaml`

     - Create secrets:

       - `python ~/AdvantEDGE/config/configure-secrets.py set ~/AdvantEDGE/config/secrets.yaml`

     - **NOTE:** Once the kuberentes secrets have been created, it is recommended to remove the `secrets.yaml` config files containing the clear-text secrets

   - Thanos:

     - `~/AdvantEDGE/config/configure-thanos-secrets.sh`

   - **NOTE:** Once the kuberentes secrets have been created, it is recommended to remove the config files containing the clear-text secrets

1. Build & Deploy AdvantEDGE:

   - Build meepctl:

     - `~/AdvantEDGE/go-apps/meepctl/install.sh`

config/.meepctl-repocfg.yaml

+96 −12
Original line number Diff line number Diff line 
version: 1.7.1

version: 1.7.2

repo:

  name: AdvantEDGE
@@ -77,10 +77,61 @@ repo: 
        # OAuth k8s secret (data: client-id, secret)

        secret: meep-oauth-gitlab



    # Metrics Configuration

    metrics:

      # Prometheus configuration

      prometheus:

        # external labels added to prometheus metrics

        external-labels:

          # deployment region (geographic or logical)

          region: etsi

          # function being monitored

          monitor: advantedge

          # prometheus environment (dev|prod)

          promenv: prod

          # unique deployment identifier

          replica: platform-ip

      # Thanos long-term storage

      thanos:

        # enable Thanos

        enabled: true

        # object store configuration secret

        secret: meep-thanos-objstore-config

        # query component

        query:

          # enable querier

          enabled: true

        # query frontend component

        query-frontend:

          # enable query frontend

          enabled: true

        # store gateway component

        store-gateway:

          # enable store gateway

          enabled: true

        # compactor component

        compactor:

          # enable compactor

          enabled: false

          # data retention times

          retention:

            # raw data retention

            resolution-raw: 30d

            # 5m downsampled data retention

            resolution-5m: 30d

            # 1h downsampled data retention

            resolution-1h: 10y

      # Thanos long-term storage archive

      thanos-archive:

        # enable Thanos archive

        enabled: true

        # archive object store configuration secret

        secret: meep-thanos-archive-objstore-config



    # Default monitoring dashboards

    dashboards:

      mec-sandbox: dashboards/mec-sandbox.json

      platform-advantedge: dashboards/platform-advantedge.json

      platform-advantedge: dashboards/platform-advantedge-thanos.json

      platform-k8s: dashboards/platform-k8s.json

      runtime-environment-node: dashboards/runtime-environment-node.json

      runtime-environment-k8s-cluster: dashboards/runtime-environment-k8s-cluster.json
@@ -616,7 +667,7 @@ repo: 
      chart: charts/influxdb

      # user supplied value file located @ .meep/user/values (use below file name)

      chart-user-values: meep-influxdb.yaml

    meep-kube-state-metrics:

    meep-ingress:

      # enable meepctl build -> deps are never built

      build: false

      # enable meepctl dockerize -> deps are never dockerized
@@ -624,10 +675,10 @@ repo: 
      # enable meepctl deploy/delete

      deploy: true

      # location of deployment chart

      chart: charts/kube-state-metrics

      chart: charts/ingress-nginx

      # user supplied value file located @ .meep/user/values (use below file name)

      chart-user-values: meep-kube-state-metrics.yaml

    meep-ingress:

      chart-user-values: meep-ingress.yaml

    meep-kube-state-metrics:

      # enable meepctl build -> deps are never built

      build: false

      # enable meepctl dockerize -> deps are never dockerized
@@ -635,20 +686,20 @@ repo: 
      # enable meepctl deploy/delete

      deploy: true

      # location of deployment chart

      chart: charts/ingress-nginx

      chart: charts/kube-state-metrics

      # user supplied value file located @ .meep/user/values (use below file name)

      chart-user-values: meep-ingress.yaml

    meep-redis:

      chart-user-values: meep-kube-state-metrics.yaml

    meep-minio:

      # enable meepctl build -> deps are never built

      build: false

      # enable meepctl dockerize -> deps are never dockerized

      dockerize: false

      # enable meepctl deploy/delete

      deploy: true

      deploy: false

      # location of deployment chart

      chart: charts/redis

      chart: charts/minio

      # user supplied value file located @ .meep/user/values (use below file name)

      chart-user-values: meep-redis.yaml

      chart-user-values: meep-minio.yaml

    meep-open-map-tiles:

      # enable meepctl build -> deps are never built

      build: false
@@ -682,6 +733,39 @@ repo: 
      chart: charts/kube-prometheus-stack

      # user supplied value file located @ .meep/user/values (use below file name)

      chart-user-values: meep-prometheus.yaml

    meep-redis:

      # enable meepctl build -> deps are never built

      build: false

      # enable meepctl dockerize -> deps are never dockerized

      dockerize: false

      # enable meepctl deploy/delete

      deploy: true

      # location of deployment chart

      chart: charts/redis

      # user supplied value file located @ .meep/user/values (use below file name)

      chart-user-values: meep-redis.yaml

    meep-thanos:

      # enable meepctl build -> deps are never built

      build: false

      # enable meepctl dockerize -> deps are never dockerized

      dockerize: false

      # enable meepctl deploy/delete

      deploy: true

      # location of deployment chart

      chart: charts/thanos

      # user supplied value file located @ .meep/user/values (use below file name)

      chart-user-values: meep-thanos.yaml

    meep-thanos-archive:

      # enable meepctl build -> deps are never built

      build: false

      # enable meepctl dockerize -> deps are never dockerized

      dockerize: false

      # enable meepctl deploy/delete

      deploy: true

      # location of deployment chart

      chart: charts/thanos

      # user supplied value file located @ .meep/user/values (use below file name)

      chart-user-values: meep-thanos-archive.yaml



  #------------------------------

  #  Packages

config/dashboards/mec-sandbox.json

+30 −30
Original line number Diff line number Diff line
@@ -34,7 +34,7 @@ 
      "type": "row"

    },

    {

      "datasource": "Prometheus",

      "datasource": "Thanos",

      "fieldConfig": {

        "defaults": {

          "custom": {},
@@ -99,7 +99,7 @@ 
      "type": "gauge"

    },

    {

      "datasource": "Prometheus",

      "datasource": "Thanos",

      "fieldConfig": {

        "defaults": {

          "custom": {},
@@ -249,7 +249,7 @@ 
      "type": "stat"

    },

    {

      "datasource": "Prometheus",

      "datasource": "Thanos",

      "fieldConfig": {

        "defaults": {

          "custom": {},
@@ -307,7 +307,7 @@ 
      "type": "stat"

    },

    {

      "datasource": "Prometheus",

      "datasource": "Thanos",

      "fieldConfig": {

        "defaults": {

          "color": {
@@ -391,7 +391,7 @@ 
      "bars": false,

      "dashLength": 10,

      "dashes": false,

      "datasource": "Prometheus",

      "datasource": "Thanos",

      "decimals": 0,

      "fieldConfig": {

        "defaults": {
@@ -490,7 +490,7 @@ 
      }

    },

    {

      "datasource": "Prometheus",

      "datasource": "Thanos",

      "fieldConfig": {

        "defaults": {

          "custom": {},
@@ -613,7 +613,7 @@ 
      "type": "text"

    },

    {

      "datasource": "Prometheus",

      "datasource": "Thanos",

      "fieldConfig": {

        "defaults": {

          "custom": {},
@@ -670,7 +670,7 @@ 
      "type": "stat"

    },

    {

      "datasource": "Prometheus",

      "datasource": "Thanos",

      "fieldConfig": {

        "defaults": {

          "custom": {},
@@ -749,7 +749,7 @@ 
      "type": "bargauge"

    },

    {

      "datasource": "Prometheus",

      "datasource": "Thanos",

      "fieldConfig": {

        "defaults": {

          "custom": {},
@@ -806,7 +806,7 @@ 
      "type": "stat"

    },

    {

      "datasource": "Prometheus",

      "datasource": "Thanos",

      "fieldConfig": {

        "defaults": {

          "custom": {},
@@ -885,7 +885,7 @@ 
      "type": "bargauge"

    },

    {

      "datasource": "Prometheus",

      "datasource": "Thanos",

      "fieldConfig": {

        "defaults": {

          "custom": {
@@ -973,7 +973,7 @@ 
      "type": "bargauge"

    },

    {

      "datasource": "Prometheus",

      "datasource": "Thanos",

      "fieldConfig": {

        "defaults": {

          "custom": {
@@ -1122,7 +1122,7 @@ 
          "type": "text"

        },

        {

          "datasource": "Prometheus",

          "datasource": "Thanos",

          "fieldConfig": {

            "defaults": {

              "custom": {},
@@ -1179,7 +1179,7 @@ 
          "type": "stat"

        },

        {

          "datasource": "Prometheus",

          "datasource": "Thanos",

          "fieldConfig": {

            "defaults": {

              "custom": {},
@@ -1258,7 +1258,7 @@ 
          "type": "bargauge"

        },

        {

          "datasource": "Prometheus",

          "datasource": "Thanos",

          "fieldConfig": {

            "defaults": {

              "custom": {},
@@ -1315,7 +1315,7 @@ 
          "type": "stat"

        },

        {

          "datasource": "Prometheus",

          "datasource": "Thanos",

          "fieldConfig": {

            "defaults": {

              "custom": {},
@@ -1394,7 +1394,7 @@ 
          "type": "bargauge"

        },

        {

          "datasource": "Prometheus",

          "datasource": "Thanos",

          "fieldConfig": {

            "defaults": {

              "custom": {
@@ -1482,7 +1482,7 @@ 
          "type": "bargauge"

        },

        {

          "datasource": "Prometheus",

          "datasource": "Thanos",

          "fieldConfig": {

            "defaults": {

              "custom": {
@@ -1635,7 +1635,7 @@ 
          "type": "text"

        },

        {

          "datasource": "Prometheus",

          "datasource": "Thanos",

          "fieldConfig": {

            "defaults": {

              "custom": {},
@@ -1692,7 +1692,7 @@ 
          "type": "stat"

        },

        {

          "datasource": "Prometheus",

          "datasource": "Thanos",

          "fieldConfig": {

            "defaults": {

              "custom": {},
@@ -1771,7 +1771,7 @@ 
          "type": "bargauge"

        },

        {

          "datasource": "Prometheus",

          "datasource": "Thanos",

          "fieldConfig": {

            "defaults": {

              "custom": {},
@@ -1828,7 +1828,7 @@ 
          "type": "stat"

        },

        {

          "datasource": "Prometheus",

          "datasource": "Thanos",

          "fieldConfig": {

            "defaults": {

              "custom": {},
@@ -1907,7 +1907,7 @@ 
          "type": "bargauge"

        },

        {

          "datasource": "Prometheus",

          "datasource": "Thanos",

          "fieldConfig": {

            "defaults": {

              "custom": {
@@ -1995,7 +1995,7 @@ 
          "type": "bargauge"

        },

        {

          "datasource": "Prometheus",

          "datasource": "Thanos",

          "fieldConfig": {

            "defaults": {

              "custom": {
@@ -2148,7 +2148,7 @@ 
          "type": "text"

        },

        {

          "datasource": "Prometheus",

          "datasource": "Thanos",

          "fieldConfig": {

            "defaults": {

              "custom": {},
@@ -2205,7 +2205,7 @@ 
          "type": "stat"

        },

        {

          "datasource": "Prometheus",

          "datasource": "Thanos",

          "fieldConfig": {

            "defaults": {

              "custom": {},
@@ -2284,7 +2284,7 @@ 
          "type": "bargauge"

        },

        {

          "datasource": "Prometheus",

          "datasource": "Thanos",

          "fieldConfig": {

            "defaults": {

              "custom": {},
@@ -2341,7 +2341,7 @@ 
          "type": "stat"

        },

        {

          "datasource": "Prometheus",

          "datasource": "Thanos",

          "fieldConfig": {

            "defaults": {

              "custom": {},
@@ -2420,7 +2420,7 @@ 
          "type": "bargauge"

        },

        {

          "datasource": "Prometheus",

          "datasource": "Thanos",

          "fieldConfig": {

            "defaults": {

              "custom": {
@@ -2508,7 +2508,7 @@ 
          "type": "bargauge"

        },

        {

          "datasource": "Prometheus",

          "datasource": "Thanos",

          "fieldConfig": {

            "defaults": {

              "custom": {

config/secrets.yaml

+4 −0
Original line number Diff line number Diff line
@@ -8,3 +8,7 @@ meep-oauth-github: 
meep-oauth-gitlab:

  client-id: "my-gitlab-client-id"

  secret: "my-gitlab-secret"



meep-minio-objstore-config:

  accesskey: "my-access-key"

  secretkey: "my-secret-key"

config/thanos-archive.yaml

0 → 100644
+12 −0
Original line number Diff line number Diff line 
type: s3

config:

  bucket: thanos-archive

  endpoint: metrics.try-mec.etsi.org

  access_key: my-access-key

  secret_key: my-secret-key

  insecure: false

  signature_version2: false

  http_config:

    idle_conn_timeout: 1m30s

    response_header_timeout: 2m

    insecure_skip_verify: false