Context Scalability extensions using CockroachDB + Removal of Stateful database inside Device + other

Lluis Gifre Renomrequested to merge
feat/context-scalability into develop

Deploy scripts:

  • Aggregated deploy scripts in folder "deploy"
  • Added deploy scripts for CockroachDB and NATS
  • Updated my_deploy.sh example deployment definition script

Manifests:

  • Migrated old GitLab.com URLs to ETSI Labs
  • Reduced default number of replicas of components to 1

GitLab CI/CD pipeline:

  • Updated and homogeneized definitions
  • Disabled deployment in Kubernetes due to changes in GitLab agents (not needed by now)

Common:

  • Added method in method_wrappers to dump component performance during unit tests
  • Updated Event handling to consider new timestamp structure in context.proto
  • Updated MockServiceImpl_Context to log reply messages
  • Updated context_queries with new method to get devices and reduced log messages
  • Updated object_factory with missing methods and corrected existing ones
  • Updated old Constants DEFAULT_CONTEXT_UUID, DEFAULT_TOPOLOGY_UUID and INTERDOMAIN_TOPOLOGY_UUID to ..._NAME
  • Updated common requirements

Context component:

  • Migrated Context to CockroachDB and NATS
  • Defined new Context ORM based on SQLAlchemy and CockroachDB engine
  • Updated context.proto and Context component to include name fields in main entities
  • Updated context.proto to include slice_ids in Context message
  • Updated unitary tests and GitLab CI/CD pipeline accordingly

Device component:

  • Removed internal stateful database and replaced by requests-replies to context
  • Extended MonitoringLoops to keep track of monitorable endpoints as an efficient in-memory database
  • Added logic to pre-load drivers for existing devices in Context when Device component starts
  • Reorganized code and separated in relevant modules and classes to improve readability
  • Updated unitary tests accordingly
  • EmulatedDriver: Reorganized code and separated in relevant modules and classes
  • EmulatedDriver: Minor updates to arrange with removal of the internal database
  • OpenConfigDriver: Minor updates to arrange with removal of the internal database

DLT component:

  • Corrected CI/CD pipeline to prevent using a network with fixed IP address ranges

Load Generator component:

  • Generalized load_generator tool to be executable as a command line and/or component
  • Added load_generator.proto to control the load_generator component
  • Added parameter to enable setting up services and slices, but not tearing them down.
  • Added parameter to generate infinite requests
  • Adapted parameter generation for services and slices to new definition of service/slice in context

Compute component:

  • Moved code to debug content of context to Compute to convert it into a debug_api NBI (by now WIP)

Monitoring component:

  • Corrected many small issues in unitary tests

PathComp component:

  • Corrected composition of requests for the Backend to be consistent with context definitions
  • Improved logging of request sent to backend
  • Corrected CI/CD pipeline to prevent using a network with fixed IP address ranges

Service component:

  • Corrected CI/CD pipeline to use real context, device, and service images; otherwise crashes

WebUI component:

  • Added basic buttons to control Load Generator component
  • Corrected logic to handle selection of context/topology
  • Added field "name" in many entities (WIP)
  • Disabled Dump ContextDB links
  • Improved listing of configuration rules of device, service and slice

Misc:

  • Many minor bug resolutions, code formatting, and cosmetic changes
  • Updated README.md file
Edited by Lluis Gifre Renom

Merge request reports

hackfest/p4/tests/Objects.py

+5 −5
Changes for hackfest/p4/tests/Objects.py: 5 added lines, 5 removed lines.
Original line number Diff line number Diff line
@@ -14,7 +14,7 @@ 


import os

from typing import Dict, List, Tuple

from common.Constants import DEFAULT_CONTEXT_UUID, DEFAULT_TOPOLOGY_UUID

from common.Constants import DEFAULT_CONTEXT_NAME, DEFAULT_TOPOLOGY_NAME

from common.tools.object_factory.Context import json_context, json_context_id

from common.tools.object_factory.Device import (

    json_device_connect_rules, json_device_emulated_connect_rules, json_device_emulated_packet_router_disabled,
@@ -28,12 +28,12 @@ from common.tools.object_factory.Topology import json_topology, json_topology_id 
from common.proto.kpi_sample_types_pb2 import KpiSampleType



# ----- Context --------------------------------------------------------------------------------------------------------

CONTEXT_ID = json_context_id(DEFAULT_CONTEXT_UUID)

CONTEXT    = json_context(DEFAULT_CONTEXT_UUID)

CONTEXT_ID = json_context_id(DEFAULT_CONTEXT_NAME)

CONTEXT    = json_context(DEFAULT_CONTEXT_NAME)



# ----- Topology -------------------------------------------------------------------------------------------------------

TOPOLOGY_ID = json_topology_id(DEFAULT_TOPOLOGY_UUID, context_id=CONTEXT_ID)

TOPOLOGY    = json_topology(DEFAULT_TOPOLOGY_UUID, context_id=CONTEXT_ID)

TOPOLOGY_ID = json_topology_id(DEFAULT_TOPOLOGY_NAME, context_id=CONTEXT_ID)

TOPOLOGY    = json_topology(DEFAULT_TOPOLOGY_NAME, context_id=CONTEXT_ID)



# ----- Monitoring Samples ---------------------------------------------------------------------------------------------

PACKET_PORT_SAMPLE_TYPES = [

manifests/cockroachdb/from_carlos/cluster-init.yaml

deleted100644 → 0
+0 −20
Changes for manifests/cockroachdb/from_carlos/cluster-init.yaml: 0 added lines, 20 removed lines.
Original line number Diff line number Diff line 
# Generated file, DO NOT EDIT. Source: cloud/kubernetes/templates/cluster-init.yaml

apiVersion: batch/v1

kind: Job

metadata:

  name: cluster-init

  labels:

    app: cockroachdb

spec:

  template:

    spec:

      containers:

      - name: cluster-init

        image: cockroachdb/cockroach:v22.1.6

        imagePullPolicy: IfNotPresent

        command:

          - "/cockroach/cockroach"

          - "init"

          - "--insecure"

          - "--host=cockroachdb-0.cockroachdb"

      restartPolicy: OnFailure

manifests/cockroachdb/README.md

+14 −5
Changes for manifests/cockroachdb/README.md: 14 added lines, 5 removed lines.
Original line number Diff line number Diff line 
# Ref: https://www.cockroachlabs.com/docs/stable/configure-cockroachdb-kubernetes.html

# CockroachDB configuration preparation



These steps reproduce how to generate Cockroach manifest files used in TeraFlowSDN and apply them to MicroK8s.

For stability reasons, we fix the versions providing the manifest files.

In future releases of TeraFlowSDN, we might consider dynamically downloading and modifying the files.



- Ref: https://www.cockroachlabs.com/docs/stable/configure-cockroachdb-kubernetes.html



## Steps:



DEPLOY_PATH="manifests/cockroachdb"

OPERATOR_BASE_URL="https://raw.githubusercontent.com/cockroachdb/cockroach-operator/master"
@@ -11,13 +19,14 @@ kubectl apply -f "${DEPLOY_PATH}/crds.yaml" 


# Deploy CockroachDB Operator

curl -o "${DEPLOY_PATH}/operator.yaml" "${OPERATOR_BASE_URL}/install/operator.yaml"

# edit "${DEPLOY_PATH}/operator.yaml"

# - add env var: WATCH_NAMESPACE='crdb'

kubectl apply -f "${DEPLOY_PATH}/operator.yaml"

nano "${DEPLOY_PATH}/operator.yaml"

# - add env var: WATCH_NAMESPACE=%TFS_CRDB_NAMESPACE%

sed s/%TFS_CRDB_NAMESPACE%/crdb/g ${DEPLOY_PATH}/operator.yaml > ${DEPLOY_PATH}/tfs_crdb_operator.yaml

kubectl apply -f "${DEPLOY_PATH}/tfs_crdb_operator.yaml"



# Deploy CockroachDB

curl -o "${DEPLOY_PATH}/cluster.yaml" "${OPERATOR_BASE_URL}/examples/example.yaml"

# edit "${DEPLOY_PATH}/cluster.yaml"

nano "${DEPLOY_PATH}/cluster.yaml"

# - set version

# - set number of replicas

kubectl create namespace crdb

.gitlab-ci.yml

+8 −7
Changes for .gitlab-ci.yml: 8 added lines, 7 removed lines.
Original line number Diff line number Diff line
@@ -16,11 +16,11 @@ 
stages:

  #- dependencies

  - build

  - build

  - test

  - unit_test

  - integ_test

  - deploy

  - funct_test

  #- deploy

  #- end2end_test



# include the individual .gitlab-ci.yml of each micro-service

include:
@@ -30,12 +30,12 @@ include: 
  - local: '/src/context/.gitlab-ci.yml'

  - local: '/src/device/.gitlab-ci.yml'

  - local: '/src/service/.gitlab-ci.yml'

  - local: '/src/dbscanserving/.gitlab-ci.yml'

  - local: '/src/opticalattackmitigator/.gitlab-ci.yml'

  - local: '/src/opticalcentralizedattackdetector/.gitlab-ci.yml'

  #- local: '/src/dbscanserving/.gitlab-ci.yml'

  #- local: '/src/opticalattackmitigator/.gitlab-ci.yml'

  #- local: '/src/opticalcentralizedattackdetector/.gitlab-ci.yml'

  - local: '/src/automation/.gitlab-ci.yml'

  - local: '/src/policy/.gitlab-ci.yml'

  - local: '/src/webui/.gitlab-ci.yml'

  #- local: '/src/webui/.gitlab-ci.yml'

  #- local: '/src/l3_distributedattackdetector/.gitlab-ci.yml'

  #- local: '/src/l3_centralizedattackdetector/.gitlab-ci.yml'

  #- local: '/src/l3_attackmitigator/.gitlab-ci.yml'
@@ -43,3 +43,4 @@ include: 
  #- local: '/src/interdomain/.gitlab-ci.yml'

  - local: '/src/pathcomp/.gitlab-ci.yml'

  #- local: '/src/dlt/.gitlab-ci.yml'

  - local: '/src/load_generator/.gitlab-ci.yml'

common_requirements.in

+14 −0
Changes for common_requirements.in: 14 added lines, 0 removed lines.
Original line number Diff line number Diff line 
# Copyright 2021-2023 H2020 TeraFlow (https://www.teraflow-h2020.eu/)

#

# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

#

#      http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.



coverage==6.3

grpcio==1.47.*

grpcio-health-checking==1.47.*