diff --git a/scripts/create_component.sh b/scripts/create_component.sh new file mode 100755 index 0000000000000000000000000000000000000000..17f6abc64140f88e0b8f09f548afdaf1f1362f6f --- /dev/null +++ b/scripts/create_component.sh @@ -0,0 +1,39 @@ +#!/bin/bash +# Copyright 2021-2023 H2020 TeraFlow (https://www.teraflow-h2020.eu/) +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +COMPONENT_NAME=$1 +PROJECTDIR=`pwd` + +mkdir -p ${PROJECTDIR}/src/${COMPONENT_NAME} +mkdir -p ${PROJECTDIR}/src/${COMPONENT_NAME}/client +mkdir -p ${PROJECTDIR}/src/${COMPONENT_NAME}/service +mkdir -p ${PROJECTDIR}/src/${COMPONENT_NAME}/tests + +touch ${PROJECTDIR}/src/${COMPONENT_NAME}/client/__init__.py +touch ${PROJECTDIR}/src/${COMPONENT_NAME}/service/__init__.py +touch ${PROJECTDIR}/src/${COMPONENT_NAME}/tests/__init__.py +touch ${PROJECTDIR}/src/${COMPONENT_NAME}/.gitlab-ci.yml +touch ${PROJECTDIR}/src/${COMPONENT_NAME}/__init__.py +touch ${PROJECTDIR}/src/${COMPONENT_NAME}/Config.py +touch ${PROJECTDIR}/src/${COMPONENT_NAME}/Dockerfile +touch ${PROJECTDIR}/src/${COMPONENT_NAME}/requirements.in + +cd ${PROJECTDIR}/src +python gitlab-ci.yml_generator.py -t latest forecaster + +cd ${PROJECTDIR}/src/${COMPONENT_NAME} +mv .gitlab-ci.yml gitlab-ci.yaml +${PROJECTDIR}/scripts/add_license_header_to_files.sh +mv gitlab-ci.yaml .gitlab-ci.yml diff --git a/tutorial/1-2-install-microk8s.md b/tutorial/1-2-install-microk8s.md index 327c6af9e477b4edd9b769371f79b32933746af6..1f1b3e6d9ac60bf8f7ecb4de29ca4a525305e9bc 100644 --- a/tutorial/1-2-install-microk8s.md +++ b/tutorial/1-2-install-microk8s.md @@ -83,17 +83,27 @@ microk8s config > $HOME/.kube/config sudo reboot ``` -## 1.2.6. Check status of Kubernetes +## 1.2.6. Check status of Kubernetes and addons +To retrieve the status of Kubernetes __once__, run the following command: ```bash microk8s.status --wait-ready ``` +To retrieve the status of Kubernetes __periodically__ (e.g., every 1 second), run the following command: +```bash +watch -n 1 microk8s.status --wait-ready +``` ## 1.2.7. Check all resources in Kubernetes +To retrieve the status of the Kubernetes resources __once__, run the following command: ```bash -microk8s.kubectl get all --all-namespaces +kubectl get all --all-namespaces ``` +To retrieve the status of the Kubernetes resources __periodically__ (e.g., every 1 second), run the following command: +```bash +watch -n 1 kubectl get all --all-namespaces +``` ## 1.2.8. Enable addons The Addons enabled are: @@ -106,10 +116,14 @@ The Addons enabled are: microk8s.enable dns hostpath-storage ingress registry ``` -__Note__: enabling some of the addons might take few minutes. - [Check status](./1-2-install-microk8s.md#124-check-status-of-kubernetes) periodically until all addons are - shown as enabled. Then [Check resources](./1-2-install-microk8s.md#125-check-all-resources-in-kubernetes) - periodically until all pods are Ready and Running. +__Important__: Enabling some of the addons might take few minutes. Do not proceed with next steps until the addons are + ready. Otherwise, the deployment might fail. To confirm everything is up and running: +1. Periodically + [Check the status of Kubernetes](./1-2-install-microk8s.md#126-check-status-of-kubernetes) + until you see the addons [dns, ha-cluster, hostpath-storage, ingress, registry, storage] in the enabled block. +2. Periodically + [Check Kubernetes resources](./1-2-install-microk8s.md#127-check-all-resources-in-kubernetes) + until all pods are __Ready__ and __Running__. ## 1.2.9. Stop, Restart, and Redeploy diff --git a/tutorial/2-2-ofc22.md b/tutorial/2-2-ofc22.md index 1a2ee8cda8c58fb534cd73f056f46345116d779e..37dfb4032d9ed09fb154ec5caf86a2199b38010c 100644 --- a/tutorial/2-2-ofc22.md +++ b/tutorial/2-2-ofc22.md @@ -2,7 +2,9 @@ This functional test reproduces the live demonstration "Demonstration of Zero-touch Device and L3-VPN Service Management Using the TeraFlow Cloud-native SDN Controller" carried out at -[OFC'22](https://www.ofcconference.org/en-us/home/program-speakers/demo/). +[OFC'22](https://ieeexplore.ieee.org/document/9748575). + + ## 2.2.1. Functional test folder @@ -33,8 +35,9 @@ To run this functional test, it is assumed you have deployed a MicroK8s-based Ku controller instance as described in the [Tutorial: Deployment Guide](./1-0-deployment.md), and you configured the Python environment as described in [Tutorial: Run Experiments Guide > 2.1. Configure Python Environment](./2-1-python-environment.md). -Remember to source the scenario settings appropriately, e.g., `cd ~/tfs-ctrl && source my_deploy.sh` in each terminal -you open. +Remember to source the scenario settings, e.g., `cd ~/tfs-ctrl && source ofc22/deploy_specs.sh` in each terminal you open. +Then, re-build the protocol buffers code from the proto files: +`./proto/generate_code_python.sh` ## 2.2.4. Access to the WebUI and Dashboard diff --git a/tutorial/2-4-ecoc22.md b/tutorial/2-4-ecoc22.md index 6fc9333b58fe7c6da51be5eefe9167853508456a..b6f92aadc692345b73c5529a4de9a56522c722d9 100644 --- a/tutorial/2-4-ecoc22.md +++ b/tutorial/2-4-ecoc22.md @@ -1,7 +1,7 @@ # 2.4. ECOC'22 Demo - Disjoint DC-2-DC L3VPN Service (WORK IN PROGRESS) This functional test reproduces the experimental assessment of "Experimental Demonstration of Transport Network Slicing -with SLA Using the TeraFlowSDN Controller" presented at [ECOC'22](https://www.ecoc2022.org/). +with SLA Using the TeraFlowSDN Controller" presented at [ECOC'22](https://www.optica.org/en-us/events/topical_meetings/ecoc/schedule/?day=Tuesday#Tuesday). ## 2.4.1. Functional test folder @@ -27,10 +27,7 @@ To run this functional test, it is assumed you have deployed a MicroK8s-based Ku controller instance as described in the [Tutorial: Deployment Guide](./1-0-deployment.md), and you configured the Python environment as described in [Tutorial: Run Experiments Guide > 2.1. Configure Python Environment](./2-1-python-environment.md). -Remember to source the scenario settings appropriately, e.g., `cd ~/tfs-ctrl && source my_deploy.sh` in each terminal -you open. -Next, remember to source the environment variables created by the deployment, e.g., -`cd ~/tfs-ctrl && source tfs_runtime_env_vars.sh`. +Remember to source the scenario settings, e.g., `cd ~/tfs-ctrl && source ecoc22/deploy_specs.sh` in each terminal you open. Then, re-build the protocol buffers code from the proto files: `./proto/generate_code_python.sh` diff --git a/tutorial/3-0-development.md b/tutorial/3-0-development.md index c2b13315aec7ee506684a3b7cc2be40b9f6407fe..c8e7d0d9eb30c4e56a76268ab9868226f3433c11 100644 --- a/tutorial/3-0-development.md +++ b/tutorial/3-0-development.md @@ -7,4 +7,6 @@ this guide assumes you are using the Oracle VirtualBox-based VM running MicroK8s ## Table of Content: - [3.1. Configure VSCode and Connect to the VM](./3-1-configure-vscode.md) - [3.2. Development Commands, Tricks, and Hints (WORK IN PROGRESS)](./3-2-develop-cth.md) -- [3.3. Debugging individual components in VSCode](./3.3-debug-comp.md) +- [3.3. Debugging individual components in VSCode](./3-3-debug-comp.md) + +- [3.X. Developing a new component: Forecaster (WORK IN PROGRESS)](./3-X-develop-new-component.md) diff --git a/tutorial/3.3-debug-comp.md b/tutorial/3-3-debug-comp.md similarity index 100% rename from tutorial/3.3-debug-comp.md rename to tutorial/3-3-debug-comp.md diff --git a/tutorial/3-X-develop-new-component.md b/tutorial/3-X-develop-new-component.md new file mode 100644 index 0000000000000000000000000000000000000000..403527335af2e66af497ee6aa4e3077e27ca5a0c --- /dev/null +++ b/tutorial/3-X-develop-new-component.md @@ -0,0 +1,132 @@ +# 3.X. Developing a new component: Forecaster (WORK IN PROGRESS) + + +## 3.X.1. Preliminary requisites +As any microservice-based architecture, the components of TeraFlowSDN can be implemented using different programming languages. +For the sake of simplicity, and given it is the most widely used programming language in TeraFlow, this tutorial page assumes the reader will use Python. + +This tutorial assumes you hace successfully completed the steps in +[2.1. Configure the Python Environment](./2-1-python-environment.md) and +[3.1. Configure VSCode and Connect to the VM](./3-1-configure-vscode.md) to prepare your environment. + + +## 3.X.2. Create the component folder structure +The source code of each component of TeraFlowSDN is hosted in a particular folder within the `src` folder. +Within that folder, typically, 3 subfolders are created: +- Folder `client`: contains a client implementation that the rest of components can use to interact with the component. + See details in [3.X.4. Create the component client](./3-X-develop-new-component.md#3x4-create-the-component-client). +- Folder `service`: contains the implementation of the service logic. + See details in [3.X.5. Create the component service](./3-X-develop-new-component.md#3x5-create-the-component-service). +- Folder `tests`: contains the set of unitary tests to be executed over the component to ensure it is properly implemented. + See details in [3.X.6. Create the component tests](./3-X-develop-new-component.md#3x6-create-the-component-tests). +- File `__init__.py`: defines the component as a sub-package of TeraFlowSDN to facilitate imports. +- File `.gitlab-ci.yml`: defines the GitLab CI/CD settings to build, test, and deploy the component in an automated manner. +- File `Config.py`: contains particular configuration settings and constants for the component. +- File `Dockerfile`: defines the recipe to construct the Docker image for the component. +- File `requirements.in`: defines the Python dependencies that are required by this component. + +You can automate the creation of this file structure running the following command. +In this example, we create the `forecaster` component. +```bash +cd ~/tfs-ctrl +scripts/create_component.sh forecaster +``` + + +## 3.X.3. gRPC Proto messages and services +The components, e.g., microservices, of the TeraFlowSDN controller, in general, use a gRPC-based open API to interoperate. +All the protocol definitions can be found in sub-folder `proto` within the root project folder. +For additional details on gRPC, visit the official web-page [gRPC](https://grpc.io/). + +In general, each component has an associated _proto_ file named as the name of the component in snake_case.proto. +For instance, the _proto_ file for the `forecaster` component being developed in this tutorial is `proto/forecaster.proto` and implements 3 RPC methods: +- `rpc GetForecastOfTopology (context.TopologyId) returns (Forecast) {}´: + Takes a topology identifier as parameter, and computes the aggregated forecast for the topology. +- `rpc GetForecastOfLink(context.LinkId) returns (Forecast) {}´: + Takes a link identifier as parameter, and computes the aggregated forecast for that link. +- `rpc CheckService (context.ServiceId) returns (ForecastPrediction) {}´: + Takes a service identifier as parameter, computes the forecast for the connections of that service, and retrieves a value indicating if the resources can support the demand. + + +## 3.X.4. Create the component client +Each component has, by default, a pre-defined client that other components can import to inter-communicate. +The client module, by default, is named as the component's name concatenated with `client`, and written in CamelCase. +For instance, the client for the `forecaster` component would be `ForecasterClient.py`. + +This file contains a class with the same name as the file, e.g., `ForecasterClient`, and implements 3 main methods, plus one method for each RPC method supported by the service. These methods are: +- Main methods: + - `__init__(host=None, port=None)`: constructor of the client class. + - `connect(self)`: triggers the connection of the client to the service pointed by host and port class parameters. + - `close(self)`: disconnects the client from the service. +- RPC methods: one for each RPC method defined in the associated service within the proto file, e.g., `proto/forecaster.proto`. + +Create file `` + + + +## 3.X.3. Connect VSCode to the VM through "Remote SSH" extension +- Right-click on "TFS-VM" +- Select "Connect to Host in Current Window" +- Reply to the questions asked + - Platform of the remote host "TFS-VM": Linux + - "TFS-VM" has fingerprint "<fingerprint>". Do you want to continue?: Continue + - Type tfs user's password: tfs123 +- You should be now connected to the TFS-VM. + +__Note__: if you get a connection error message, the reason might be due to wrong SSH server fingerprint. Edit file + "<...>/.ssh/known_hosts" on your local user account, check if there is a line starting with + "[127.0.0.1]:2200" (assuming previous port forwarding configuration), remove the entire line, save the file, + and retry connection. + + +## 3.X.4. Add SSH key to prevent typing the password every time +This step creates an SSH key in the VM and installs it on the VSCode to prevent having to type the password every time. + +- In VSCode (connected to the VM), click menu "Terminal > New Terminal" +- Run the following commands on the VM's terminal through VSCode +```bash +ssh-keygen -t rsa -b 4096 -f ~/.ssh/tfs-vm.key + # leave password empty +ssh-copy-id -i ~/.ssh/tfs-vm.key.pub tfs@10.0.2.10 + # tfs@10.0.2.10's password: <type tfs user's password: tfs123> +rm .ssh/known_hosts +``` + +- In VSCode, click left "Explorer" panel to expand, if not expanded, and click "Open Folder" button. + - Choose "/home/tfs/" + - Type tfs user's password when asked + - Trust authors of the "/home/tfs [SSH: TFS-VM]" folder when asked +- Right click on the file "tfs-vm.key" in the file explorer + - Select "Download..." option + - Download the file into your user's accout ".ssh" folder +- Delete files "tfs-vm.key" and "tfs-vm.key.pub" on the TFS-VM. + +- In VSCode, click left "Remote Explorer" panel to expand + - Click the "gear" icon next to "SSH TARGETS" on top of "Remote Explorer" bar + - Choose to edit "<...>/.ssh/config" file (or equivalent) + - Find entry "Host TFS-VM" and update it as follows: +``` +Host TFS-VM + HostName 127.0.0.1 + Port 2200 + ForwardX11 no + User tfs + IdentityFile "<path to the downloaded identity private key file>" +``` + - Save the file +- From now, VSCode will use the identity file to connect to the TFS-VM instead of the user's password. + + +## 3.X.5. Install VSCode Python Extension (in VSCode server) +This step installs Python extensions in VSCode server running in the VM. + +- In VSCode (connected to the VM), click left button "Extensions" +- Search "Python" extension in the extension Marketplace. +- Install official "Python" extension released by Microsoft. + - By default, since you're connected to the VM, it will be installed in the VSCode server running in the VM. + +- In VSCode (connected to the VM), click left button "Explorer" +- Click "Ctrl+Alt+P" and type "Python: Select Interpreter". Select option "Python: 3.9.13 64-bit ('tfs')" + +in terminal: set python path to be used by VSCode: +`echo "PYTHONPATH=./src" > ~/tfs-ctrl/.env`