Merge branch 'STF625' into 'master'

  - [Integrating the new microservice in AdvantEDGE](#integrating-the-new-microservice-in-advantedge)

  - [Network Scenario Creation](#network-scenario-creation)

  - [Frontend Enhancements](#frontend-enhancements)

  - [Upgrading an existing MEC Service](#upgrading-an-existing-mec-service)

This guide contains all the steps that need to be taken for introducing a MEC service as a new feature in the MEC Sandbox frontend and the AdvantEDGE backend.

- Swagger UI integration

- Grafana dashboard for monitoring service-related metrics

See [Frontend Enhancement](./frontend-enhancements.md) for details on how to integrate the new feature into AdvantEDGE.

See [Frontend Enhancement](./frontend-enhancements.md) for details on how to upgrade the MEC Sandbox frontend for the new feature.

## Upgrading an existing MEC Service

The above sections are strictly related to adding a new service in the MEC Sandbox. However, there may be a need for updating an existing MEC service in the Sandbox as part of feature enhancements. 

In that case, there are some aspects that need to be taken into consideration while upgrading an existing MEC Service API in MEC Sandbox for both frontend and backend.

See [Service Upgradation Guide](./service-upgradation.md) for details on how to upgrade an existing MEC service.

After having restructured the client & server stubs and developing minimal server-side & client-side logic for the new feature using the [development guide](./development-guide.md), move the code files into AdvantEDGE repository or its fork using the following guidelines:

- [Server side](./development-guide.md#server-side) code contained in a separate folder named `meep-<mec-identifier>` will go under `AdvantEDGE/go-apps/` directory

- [Server side (For multiple APIs)](./development-guide.md#server-side-for-multiple-apis) code contained in a separate folder named `meep-<mec-identifier>` will go under `AdvantEDGE/go-apps/` directory

- [Client side](./development-guide.md#client-side) code contained in a folder named `meep-<mec-identifier>-client` will be placed under `AdvantEDGE/go-packages/` directory

- [Testing](./development-guide.md#unit-tests-for-the-new-micro-service) code... (**TODO** _Info about test files to be added later._)

### Changes in AdvantEDGE `.meepctl-repocfg.yaml`

The changes that need to be made to `.meepctl-repocfg.yaml` file are:

- Add the new service in `repo.sandbox.go-apps` section

- Add the new service in `repo.sandbox.go-apps` section. In case if a service contains multiple APIs, then rest of the lines will be same. Just add the corresponding APIs information under _api_ tag, as mentioned below:

  ```

  api:

  - name: 'AdvantEDGE MEC <api_name_1>'

    file: go-apps/meep-<mec-identifier>/api/<api_tag_1>/swagger.yaml

  - name: 'AdvantEDGE MEC <api_name_2>'

    file: go-apps/meep-<mec-identifier>/api/<api_tag_2>/swagger.yaml

  ...

  ...

  ```

- To allow the `meep-virt-engine` service to deploy the new service's pod, the service should be added to the following sections:

  - `repo.sandbox.go-apps.meep-virt-engine.docker-data`

  - `repo.sandbox.go-apps.meep-virt-engine.sandbox-pods`

  >**NOTE:** These settings will enable the `meep-virt-engine` to deploy the new service by default upon each user sandbox creation. For configuring the new service to only get deployed upon selection of a certain network scenrio, see [Network Scenarios](./network-scenarios.md#adding-a-new-service-in-an-existing-network-scenario).

  >**NOTE:** These settings will enable the `meep-virt-engine` to deploy the new service by default upon each user sandbox creation. For configuring the new service to only get deployed upon selection of a certain network scenario, see [Network Scenarios](./network-scenarios.md#adding-a-new-service-in-an-existing-network-scenario).

- Add client for the new service in `repo.packages.go-packages` section (if one is available/required)

- Add information for the new service in the monitoring engine section `repo.sandbox.go-apps.meep-mon-engine.sandbox-pods`

Additionally for making the Swagger UI available in the Sandbox frontend for the new service, these steps need to be taken:

1. Copy the Swagger or OpenAPI specifications of the new MEC Service from backend (available under `mec-sandbox/go-apps/meep-<mec-identifier>/api/swagger.yaml`) to `mec-sandbox/config/api/` folder and rename the file from `swagger.yaml` to `<mec-identifier>-api.yaml`. Additionally

1. Copy the Swagger or OpenAPI specifications of the new MEC Service from backend (available under `AdvantEDGE/go-apps/meep-<mec-identifier>/api/swagger.yaml`) to `mec-sandbox/config/api/` folder and rename the file from `swagger.yaml` to `<mec-identifier>-api.yaml`.

    > **Note:** In case of multiple APIs, copy each Swagger or OpenAPI specifications of the new MEC Service from the backend (available under `AdvantEDGE/go-apps/meep-<mec-identifier>/api/<api_tag_*>/swagger.yaml`) to the `mec-sandbox/config/api/<api_tag_*>/` subfolder.

2. After performing the above, do the following:

    - Customize the YAML to hide unnecessary endpoints in the UI

    - Add necessary license description in the `info` section

2. Add this location of OAS yaml in the `repo.sandbox.go-apps.<mec-identifier>.user-api` section of `.meepctl-repocfg.yaml` file

3. Add this location of OAS yaml in the `repo.sandbox.go-apps.<mec-identifier>.user-api` section of `.meepctl-repocfg.yaml` file

In addition to `.meepctl-repocfg.yaml` file, the following files will need to be updated when creating a new micro-service:

- `mec-sandbox/config/long-term-storage-repocfg.yaml`

All the Helm charts for multiple services are available under the `AdvantEDGE/charts` folder. Once a new service is added, its Helm chart needs to be added in this folder under the name `meep-<mec-identifier>`.

The template for the Helm chart for the new service can be obtained from any other micro-service like `meep-rnis`, `meep-wais` etc. The service name in each of the files for the template will need to be changed to reflect the name of the new service.

The template for the Helm chart for the new service can be obtained from any other micro-service like `meep-rnis`, `meep-wais` etc. The service name in each of the files for the template will need to be changed to reflect the name of the new service. In case of multiple APIs scenario, `meep-tm` charts can be consulted.

## _meep-applications_ Configuration

To make the new service be properly recognizable by the MEC Sandbox frontend, it will need to be added in the `SysAppNames` variable in `AdvantEDGE/go-packages/go-applications/application.store.go` file. This will make sure that the new service is recognized as a service and not as an application instance.

To make the new service be properly recognizable by the MEC Sandbox frontend, it will need to be added in the `SysAppNames` variable in `AdvantEDGE/go-packages/meep-applications/application.store.go` file. This will make sure that the new service is recognized as a service and not as an application instance.

## Upgrade MEC Sandbox

After the correct network scenario is selected in the MEC Sandbox UI, make sure that the new MEC service is not being shown in the [box](../Sandbox-User-Interface/README.md#sbox04) under _MEC App. Instance IDs_.

### 3. Communication with the common message bus

Check the logs of the container running the microservice and see if it is sending and receving messages on the common message bus. If there is no logic for sending messages through SBI, at least the service should be receiving periodic "SCENARIO-UPDATE" messages for different events related to UE mobility etc.

Check the logs of the container running the microservice and see if it is sending and receiving messages on the common message bus. If there is no logic for sending messages through SBI, at least the service should be receiving periodic "SCENARIO-UPDATE" messages for different events related to UE mobility etc.

### 4. Successful responses from the service API

Send a request to one of the exposed endpoints of the new service using _curl_, _Postman_ or any other tool to verify that the service is returning valid responses through its NBI accroding to the developed emulation logic.

Send a request to one of the exposed endpoints of the new service using _curl_, _Postman_ or any other tool to verify that the service is returning valid responses through its NBI according to the developed emulation logic.

### 5. Swagger UI for the new service

Follow these steps to validate successful Swagger UI generation for the new service:

    - [Client SDK](#client-sdk)

  - [Development of Emulation Logic](#development-of-emulation-logic)

    - [Server-side](#server-side)

    - [Server-side (For multiple APIs)](#server-side-for-multiple-apis)

    - [Client-side](#client-side)

      - [Code organization](#code-organization)

  - [Unit tests for the new micro-service](#unit-tests-for-the-new-micro-service)

  - [Integration tests for the new micro-service](#integration-tests-for-the-new-micro-service)

    - [Executing a test campaign](#executing-a-test-campaign)

  - [System tests for the new micro-service](#system-tests-for-the-new-micro-service)

  - [Adding Documentation for an API/Service](#adding-documentation-for-an-apiservice)

  - [Dockerization](#dockerization)

    - [Creating a Dockerfile](#creating-a-dockerfile)

- [Useful K8s commands](#useful-k8s-commands)

This guide gives a high-level overview of all the steps that need to be taken for introducing a new MEC Service in the MEC Sandbox. It documents the overall development-related aspects that need to be covered while introducing a new service in the AdvantEDGE backend. The integration aspects for feature-development are listed in [Backend Integration](./backend-integration.md).

   ```

   Here `swagger-codegen-cli.jar` is the downloaded Swagger Codegen _jar_ file, `oas-file.yaml` is the OAS YAML file and `~/output` is the path to the folder where the server stubs will be generated to.

> **Note:** Both server stubs and client SDK are generated using Swagger Codegen. The Go client SDK is required to develop the testing tool for the MEC micro-service (see clause [creating test code](#unit-tests-for-the-new-micro-service)). The Go client SDK is required to develop the backend of a MEC application using MEC030 micro-service.

> **Note:** Both server stubs and client SDK are generated using Swagger Codegen. The Go client SDK is required to develop the testing tool for the MEC micro-service (see clause [creating test code](#unit-tests-for-the-new-micro-service)).

## Development of Emulation Logic

   Run the following commands to create the _go.mod_ and _go.sum_ files:

   ```sh

   go mod init github.com/InterDigitalInc/AdvantEDGE/go-apps/meep-<mec-identifier>

   go get github.com/InterDigitalInc/AdvantEDGE/go-apps/meep-<mec-identifier>

   go mod tidy

   ```

After doing the above steps, the files will look something like the file/folder structure shown below:

            \____ <mec-identifier>.go

```

### Server-side (For multiple APIs)

After the server stubs have been generated for each APIs separately (see [server stubs](#server-stubs)), the folder structure for single API will be something like the structure shown below:

```

meep-<mec-identifier>

     |

     \____ main.go

     |

     \____ api

     |      |

     |      \____ swagger.yaml

     |

     \____ go

            |

            \____ api_<api_tag_*>.go

            |

            \____ api_unsupported.go

            |

            \____ logger.go

            |

            \____ model_<model_name>.go

            |

            \____ README.md

            |

            \____ routers.go

```

There may be multiple files for `api_<api_tag_*>` and `model_<model_name>` depending on the tags and component schemas in the OAS file used to generate the server stubs.

> Note: _*_ in _api_tag_*_ used to indicate any single API.

Steps to implement server side emulation logic for multiple APIs:

1. **Creating new folder structure**

   The folder structure for multiple APIs will be something like the structure shown below:

```

meep-<mec-identifier>

     |

     \____ main.go

     |

     \____ api

     |      |

     |      \____ <api_tag_1>

     |      |              |

     |      |              \____swagger.yaml

     |      |____ <api_tag_2>

     |                     |

     |                     \____swagger.yaml

     |

     \____ server

               |

               \____<api_tag_1>

               |             |

               |             \____ api_<api_tag_1>.go

               |             |

               |             \____ api_unsupported.go

               |             |

               |             \____ model_<model_name>.go

               |             |

               |             \____ README.md

               |

               \____<api_tag_2>

               |              |

               |              \____ api_<api_tag_2>.go

               |              |

               |              \____ api_unsupported.go

               |              |

               |              \____ model_<model_name>.go

               |              |

               |              \____ README.md

               |

               |

               \____ routers.go

               |

               \____ logger.go

```

2. **Copying _API_ Folder**

   From the generated server stubs for each API, copy the `api/swagger.yaml` file to the corresponding `api/<api_tag_*>` sub-folder for multiple APIs scenario.

> Note: _*_ in _api_tag_*_ used to indicate any single API.

3. **Copying _Go_ Folder**

   For multiple APIs scenario, create sub-folders with specific `<api-tag_*>` name under `server` folder for each API. Then copy `api_<api_tag_*>.go`, `api_unsupported.go` (if any), `model_<model_name>.go` and `README.md` files from each API `go` folder to its specific `<api-tag_*>` folder.

4. **Modify _main.go_**

   For multiple API scenario, there should be a single _main.go_ file to initialize all the corresponding APIs for a particular service. The following changes should be done in _main.go_ file:

   - _main()_: main function to call the other functions to initialize the service

      - Add an anonymous function inside the _main()_ function which will initialize the service

      - Add another anonymous function inside the _main()_ function which will initialize the metrics endpoint

   **NOTE:** Observe the _main.go_ files in existing AdvantEDGE / Sandbox services to validate your code.

5. **Copy and Modifying Routers**

   There should be a single _routers.go_ file under _server_ folder in case of multiple APIs. From the generated server stubs, copy `router.go` file from a single API and paste it in the `server/routers.go` file. Now copy the routes defined in rest of the APIs in the same `server/routers.go` file.

   Now, in the `server/routers.go` file, make the following changes:

   - Add new handler for `/api` and `/user-api` endpoints for Swagger UI

   - Remove `/sandboxname` before each endpoint in the _routes_ if it appears

   - Import the `AdvantEDGE/go-packages/meep-metrics` package and add the `MetricsHandler()` handler function for the metrics for Grafana dashboard.

   > Other MEC services in AdvantEDGE can be referred to for implementing the above changes.

6. **Create the service handler file**

   Create `server/<mec-identifier>.go` and `server/<api_tag_*>/<api_tag_*>.go` files. The `server/<mec-identifier>.go` file will contain all the common handler functions for endpoints and the emulation logic for all the APIs in that service. And `server/<api_tag_*>/<api_tag_*>.go` file will contain all the handler functions for endpoints and the emulation logic for that specific service API.

7. **Implement handler function for unsupported endpoints**

   In the `server/<api_tag_*>/api_unsupported.go` file (if exists), for each of the handler functions, replace the previous code with `notImplemented(w, r)` and add this function in the `server/<mec-identifier>.go` as follows:

   ```go

   func notImplemented(w http.ResponseWriter, r *http.Request) {

      w.Header().Set("Content-Type", "application/json; charset=UTF-8")

      w.WriteHeader(http.StatusNotImplemented)

   }

   ```

8. **Separate handler function(s) for the endpoints**

   Remove the previous code in each of the `server/<api_tag_*>/api_<api_tag_*>.go` file and replace it with a handler function. Implement all those handler functions in the `server/<mec-identifier>.go` file.

9. **Implement SBI Communication Logic (If required)**

   The new MEC service will need to communicate with other AdvantEDGE/Sandbox services to, for example, get updated scenario information via South Bound Interface (SBI). To do this, create a `server/sbi` folder and create a `<mec-identifier>-sbi.go` file in it. Implement all the communication and initialization related logic of SBI in this file. This will include communication with the message queue and Redis cache from where it will obtain the necessary information.

   Initialization of the SBIs and logic for receiving information from them will need to be implemented in the `server/<mec-identifier>.go` file. Communication with SBI will be initiated as soon as the MEC service (or service pod) is up and running.

   > Other MEC services in AdvantEDGE can be referred to for implementing the SBI communication logic.

10. **Creating _go.mod_ and _go.sum_ files**

   Run the following commands to create the _go.mod_ and _go.sum_ files:

   ```sh

   go mod init github.com/InterDigitalInc/AdvantEDGE/go-apps/meep-<mec-identifier>

   go mod tidy

   ```

After doing the above steps, the files will look something like the file/folder structure shown below:

```

meep-<mec-identifier>

     |

     \____ main.go

     |

     \____ go.mod

     |

     \____ go.sum

     |

     \____ sbi

     |      |

     |      \____ <mec-identifier>-sbi.go

     |

     \____ api

     |      |

     |      \____ <api_tag_1>

     |      |              |

     |      |              \____swagger.yaml

     |      |____ <api_tag_2>

     |                     |

     |                     \____swagger.yaml

     \____ server

               |

               \____<api_tag_1>

               |              |

               |              \____ api_<api_tag_1>.go

               |              |

               |              \____ api_unsupported.go

               |              |

               |              \____ <api_tag_1>.go

               |              |

               |              \____ model_<model_name>.go

               |              |

               |              \____ README.md

               |

               \____<api_tag_2>

               |              |

               |              \____ api_<api_tag_2>.go

               |              |

               |              \____ api_unsupported.go

               |              |

               |              \____ <api_tag_2>.go

               |              |

               |              \____ model_<model_name>.go

               |              |

               |              \____ README.md

               |

               \____ <mec-identifier>.go

               |

               \____ routers.go

               |

               \____ logger.go

```

### Client-side

During the phase of code generation (see [code generation](#generating-server-and-client-side-code)), two stubs of code are generated:

1. The server code (see clause [Server stubs](#server-stubs))

     |

     \____ <mec-identifier>.go

```

## Unit tests for the new micro-service

Unit tests should be written for a MEC micro-service for its validation. The steps to achieve this are described below:

Unit tests should be written for a MEC micro-service for its validation. The purpose of the Unit tests is to provide one or more testing function for each of the exposed API functions.

The steps to achieve this are described below:

1. **Create a testing file**

   In the server stubs folder, execute the following command:

   ```sh

   # Launch the testing DBs...

   ~/AdvantEDGE/test/start-ut-env.sh

   # Do the tests gain and again...

   $ go test -count=1 ./... -cover

   # When tests are achieved, stop the DBs

   ~/AdvantEDGE/test/stop-ut-env.sh

   ```

## Integration tests for the new micro-service

The script `run-ut.sh` executes all the tests against the different micro-services and components.

The script `stop-ut-env.sh` deletes the components previously started by `start-ut-env.sh` script.

Note: The following code executes all the tests against the different micro-services and components and store standard and error outputs into a file 'run_out.txt'

```sh

cd test

./start-ut-env.sh

./run-ut.sh 2>&1 | tee run_out.txt

./stop-ut-env.sh

```

## System tests for the new micro-service

System tests should be written for a MEC micro-service for its validation. The purpose of System tests is to test the micro service logic (functional tests).

The steps to achieve this are described below:

1. **Create a testing files**

   In the folder ~/AdvantEDGE/test/system, create two files;

   - The source code file, named `<mec-identifier>_test.go` (e.g. vis_test.go),

   - The network scenario file, named `<mec-identifier>-system-test.yaml` (e.g. vis-system-test.yaml).

The network scenario is used by the MEC micro-service testing functions.

2. **Implement System tests**

   At least, the following functions shall be implemented into `<mec-identifier>_test.go` file:

   - The initialization functions: `func init()` and `func initialiseRnisTest()`

   - The cleanup function: `func clearUpRnisTest()`

   - load scenario testing function: `func Test_RNIS_load_scenarios(t *testing.T)`

3. **Executing the tests for a single MEC service**

   Execute the following command:

   ```sh

   # Launch the AdvantEDGE platform...

   meepctl deploy dep && sleep 5 && meepctl deploy core && sleep 5

   # Launch the test campaign...

   $ cd ~/AdvantEDGE/test/system

   $ ./loopTest.sh 2

   # When tests are achieved, stop the AdvantEDGE platform

   meepctl delete core && sleep 5 && meepctl delete dep && sleep 5

   ```

## Adding Documentation for an API/Service

After a new service is created, its documentation need to be added in the *AdvantEDGE/docs* folder. This documentation can be generated from the OpenAPI Specifications of the service/API:

1. Install Java on your system

2. Download the OpenAPI Generator v5.0.0-beta2 _jar_ file:

   ```sh

   $ wget https://repo1.maven.org/maven2/org/openapitools/openapi-generator-cli/5.0.0-beta2/openapi-generator-cli-5.0.0-beta2.jar -O openapi-generator-cli.jar

   ```

3. Run the following command to generate documentation as markdown files:

   ```sh

   $ java -jar openapi-generator-cli.jar generate \

     -i oas-file.yaml \

     -g markdown \

     -o ~/output

   ```

   Here `openapi-generator-cli.jar` is the downloaded OpenAPI Generator _jar_ file, `oas-file.yaml` is the OAS YAML file and `~/output` is the path to the folder where the documentation will be generated to.

4. Copy the contents of the `output` folder over to the `AdvantEDGE/docs/api-<mec-identifier>` folder.

## Dockerization

The new feature (MEC service) will run inside a pod as a micro-service in AdvantEDGE backend. Therefore, we need to create a Dockerfile to containerize the new MEC service.

```

As a next step, code included in the directory structure above will be added into AdvantEDGE's fork repository. See [Code Integration](./backend-integration.md#code-integration) for more details.

# Useful K8s commands

This clause provide a set of useful K8s command lines for debugging micro-service and demo applications.

The commands (and their outputs) are just examples:

- <i>demo4sb</i> is the Sandbox named

- <i>meep-dai</i> is the DAI (MEC-016) micro-services.

They have to be modified based of the existing list of pods.

- To retrieve list of pods

```bash

$ kubectl get pods --all-namespaces

```

- To retrieve pod descriptions such as internal IP address, Volumes...

```bash

$ kubectl describe pod meep-<mec_identifier>-<deploy_id>-<pod_id> -n <namespace>

```

- To retrieve pod logs into a text file named out.txt

```bash

$ kubectl logs meep-<mec_identifier>-<deploy_id>-<pod_id> -n <namespace> > out.txt

```

- To retrieve pod's container logs into a text file named out.txt

```bash

$ kubectl logs meep-<mec_identifier>-<deploy_id>-<pod_id> <container_name> -n <namespace> > out.txt

```