fix for #29 (updated with new k8s model concept, but figures)

**CAMARA API Usage - Service Operation**:

- The API request includes QoD (Quality on Demand) parameters and is accessible via a specified NodePort and/or endpoint URL (e.g. `portal.osl/{serviceAUUID}/camaraapiFQDN`).

- A client performs a CAMARA QoD Request.

- The API request includes CAMARA QoD Provisioning parameters and is accessible via a specified NodePort and/or endpoint URL (e.g. `portal.osl/{serviceAUUID}/camaraapiFQDN`).

- A client performs a CAMARA QoD Provisioning request.

- An event is created and circulated within OSL message bus to notify about the request.

- The event results into a TMF request that is used to patch and customize the Network Operator's Profile enforcer service in the Service Inventory.

- The configuration is passed through OSL Orchestrator and CRIDGE down to the `5G Core Controller`.

# Quality on Demand API Proof of Concept

# Quality on Demand (QoD) Provisioning API Proof of Concept

## Candidate CAMARA API - QoD Provisioning

Additionally, you may create a characteristic titled “*camaraAPI.results*”, which you can use to have visibility of the QoD Provisionings processed by the API, at OSL level. Still, this characteristic is not required.

Therefore, OSL’s CAMARAaaS (QoD API exposure service) will contain the following TMF Service Characteristics:

Therefore, OSL’s CAMARAaaS (QoD Provisioning API exposure service) will contain the following TMF Service Characteristics:

- *messageBroker.address*

- *messageBroker.port*

These results are then processed by the *CamaraResultsProcessor* class, which updates each provisioning accordingly in the database.

## General concepts of the QoD service (the 5G Core Provider/Operator Service) as OSL TMF Service Specification

## General concepts of the QoD Provisioning service (the 5G Core Provider/Operator Service) as OSL TMF Service Specification

Having chosen the candidate API, the first step is to find a way to map the possible requests to TMF Service characteristics of the Operator’s service. Our approach in general is the following:

- Design the QoD service that will accept the necessary QoD related characteristics

- Implement the QoD service as a controller able to manage your 5G Core

- Design the QoD Provisioning service that will accept the necessary QoD Provisioning related characteristics

- Implement the QoD Provisioning service as a controller able to manage your 5G Core

- Deploy it in a kubernetes cluster that OSL can manage (via CRIDGE)

By looking at API’s specification, it is clear that at least 3 operations are required: (i) the creation of a QoD profile, (ii) its deletion, and (iii) listing all active QoD profiles. Therefore, we can proceed with evaluating the payload required for creating a QoD Provisioning. This payload involves various fields, which can be translated to the TMF Service Characteristics:

By looking at API’s specification, it is clear that at least 3 operations are required: (i) the creation of a QoD Provisioning, (ii) its deletion, and (iii) listing all active QoD Provisionings. Therefore, we can proceed with evaluating the payload required for creating a QoD Provisioning. This payload involves various fields, which can be translated to the TMF Service Characteristics:

- *qodProv.device.phoneNumber*

- *qodProv.device.networkAccessIdentifier*

- *qodProv.sink*

- *qodProv.sinkCredential.credentialType*

In order to support interaction with the previously defined OSL’s CAMARAaaS and alignment the CAMARA QoD Provisioning API models in general, the QoD service must be designed, at least, with these characteristics. 

In order to support interaction with the previously defined OSL’s CAMARAaaS and alignment the CAMARA QoD Provisioning API models in general, the QoD Provisioning service must be designed, at least, with these characteristics. 

Still, since there are various operations that can take place (CREATE and DELETE), a characteristic is also needed to map this. Therefore, the Operator's QoD service must also have a characteristics titled *qodProv.operation*. The DELETE operation is achieved based on a provisioning Id, and therefore the respective characteristics is needed: *qodProv.provisioningId.*

Still, since there are various operations that can take place (CREATE and DELETE), a characteristic is also needed to map this. Therefore, the Operator's QoD Provisioning service must also have a characteristics titled *qodProv.operation*. The DELETE operation is achieved based on a provisioning Id, and therefore the respective characteristics is needed: *qodProv.provisioningId.*

Finally, characteristic is required to store the provisionings that were enforced by the Operator's QoD service. We can define this characteristic as *camaraResults*.

Finally, characteristic is required to store the provisionings that were enforced by the Operator's QoD Provisioning service. We can define this characteristic as *camaraResults*.

Therefore, for an Operator’s service to be controlled by OSL’s CAMARAaaS specification, it needs to be designed with, at least, the following characteristics:

- *qodProv.provisioningId*

- *camaraResults*

Additional characteristics are fully supported. Those can be custom characteristics that are required by the Operator's QoD Service.

Additional characteristics are fully supported. Those can be custom characteristics that are required by the Operator's QoD Provisioning Service.

In regard to the *camaraResults* characteristic, to allow interoperability, it must store a Stringified JSON Array with the enforced QoD Provisionings. **The schema of each provisioning should be the one defined in CAMARA’s QoD Provisioning API Specification.** 

## How To / Demonstration

### 1. Dummy QoD Kubernetes Operator Service Design

### 1. Dummy QoD Provisioning Kubernetes Operator Service Design

For this prototype, since we cannot deliver a QoD Service implementation for a specific commercial core, we created a Dummy QoD Kubernetes Operator Service which emulates the 5G Core configuration. In future releases of OSL we will offer solutions for some open source 5G cores, and/or some code templates to build your own 5G Core Operator.

For this prototype, since we cannot deliver a QoD Provisioning Service implementation for a specific commercial core, we created a Dummy QoD Provisioning Kubernetes Operator Service which emulates the 5G Core configuration. In future releases of OSL we will offer solutions for some open source 5G cores, and/or some code templates to build your own 5G Core Operator.

We will start by looking at the Dummy QoD Kubernetes Operator Service we have created to demonstrate this Add-on. The Service will be offered as a simple Custom Resource (CR) deployed in a Kubernetes cluster. You may find its Custom Resource Definition (CRD) under `/DummyOperatorService/crd.yaml` Look at the CRD fields. Please notice that these were defined according with what we introduced in the previous section.

We will start by looking at the Dummy QoD Provisioning Kubernetes Operator Service we have created to demonstrate this Add-on. The Service will be offered as a simple Custom Resource (CR) deployed in a Kubernetes cluster. You may find its Custom Resource Definition (CRD) under `/DummyOperatorService/crd.yaml` Look at the CRD fields. Please notice that these were defined according with what we introduced in the previous section.

The first step is then to install this CRD in your kubernetes cluster. To this end, you may use the following command: `make create-dummy-operator-crd`

Regarding the RFS Service, you must set the following characteristics:

- _CR_CHECKVAL_AVAILABLE = RUNNING

- _CR_CHECK_FIELD = spec.status

- _CR_CHECKVAL_HEALTH_UP = RUNNING

- _CR_CHECKVAL_HEALTH_PENDING = NOT_RUNNING

By setting these characteristics, you will rely on the value of `spec.status` to set the service as `active`. Ideally, the 5G Core provider would have implemented a proper Kubernetes Operator for this Custom Resource to implement the requested QoD. However, as discussed already, for demonstration purposes, we will short-circuit the behavior and we will set `spec.status` to `RUNNING` immediately after deployment.

By setting these characteristics, you will rely on the value of `spec.status` to set the service as `active`: `RUNNING` marks the resource as healthy (UP), while `NOT_RUNNING` (the CRD default) keeps it pending. Optionally, you may also set `_CR_CHECKVAL_AVAILABLE = RUNNING` to reflect the resource status in the Resource Inventory (bookkeeping only). See [Mapping the CR lifecycle](../../service_design/kubernetes/exposing_kubernetes_resources.md#mapping-the-cr-lifecycle-that-is-defined-in-the-crd-with-the-openslice-tmf-based-resource-lifecycle) for all available mappings.

Ideally, the 5G Core provider would have implemented a proper Kubernetes Operator for this Custom Resource to implement the requested QoD. However, as discussed already, for demonstration purposes, we will short-circuit the behavior and we will set `spec.status` to `RUNNING` immediately after deployment.

Then, you can proceed to create a Customer-Facing-Service (CFS) Specification, which will incorporate the just created RFS Service. More information is available at: [Exposing Kubernetes Operators as a Service : Offering "Calculator as a Service" through OpenSlice](../../service_design/examples/calculator_crd_aas/calculator_crd_aas.md). To create the CFS Service's characteristics, you may use the Service Specification available at `/DummyOperatorService/OSLArtifacts/DummyOperatorService-CFS-Specification.json` . You may manually create the CFS Service, or you may onboard this Service Specification by making a POST request to `{{url}}/tmf-api/serviceCatalogManagement/v4/serviceSpecification`.

#### 2.1 OSL CAMARAaaS QoD Provisioning API - Kubernetes Operator

The previous docker image shall make available the CAMARA QoD Provisioning API. However, these API will be made available through Custom Resources of Type `CAMARAaaS-QoDProvisiongAPI`. Therefore, we also need a Kubernetes Operator to manage these resources. The Operator’s code can be found under `/QoDProvisioningAPI/Operator`. There, you have the source code of the Kubernetes Operator. The Operator's Helm Chart uses the default OSL registry docker images. If you are interesting in developing your own, please refer to the Makefile. To install the Operator, run the following command: 

The previous docker image shall make available the CAMARA QoD Provisioning API. However, this API will be made available through Custom Resources of Type `CAMARAaaS-QoDProvisiongAPI`. Therefore, we also need a Kubernetes Operator to manage these resources. The Operator’s code can be found under `/QoDProvisioningAPI/Operator`. There, you have the source code of the Kubernetes Operator. The Operator's Helm Chart uses the default OSL registry docker images. If you are interested in developing your own, please refer to the Makefile. To install the Operator, run the following command: 

```bash

helm install camaraaas-qod-prov-operator ./QoDProvisioningAPI/Operator/chart --namespace <namespace_where_the_operator_shall_be_deployed> --create-namespace

After these updates, create the CR by running the command: `make create-operator-test-cr`.

When the CR is created, its operator will deploy the CAMARA QoD API in a pod and expose it via a K8s Node Port. The URL where the API is available is published under the CR field `spec.camaraAPI.url` (e.g. [http://10.255.28.73:32630](http://10.255.28.73:32630/)). Check this field by running `make describe-operator-test-cr`. To confirm the API is running, access *<URL>/docs*. You should see the following:

When the CR is created, its operator will deploy the CAMARA QoD Provisioning API in a pod and expose it via a K8s Node Port. The URL where the API is available is published under the CR field `spec.camaraAPI.url` (e.g. [http://10.255.28.73:32630](http://10.255.28.73:32630/)). Check this field by running `make describe-operator-test-cr`. To confirm the API is running, access *<URL>/docs*. You should see the following:

![QoD Provisioning API Docs](./images/QoDProvisioningAPI-Docs.png)

Regarding the RFS Service, you must set the following characteristics:

- _CR_CHECKVAL_AVAILABLE = RUNNING

- _CR_CHECK_FIELD = spec.camaraAPI.status

- _CR_CHECKVAL_HEALTH_UP = RUNNING

- _CR_CHECKVAL_HEALTH_PENDING = NOT_RUNNING

By setting this characteristics, you will rely on the value of `spec.camaraAPI.status` to set the service as `active`. The previous operator, when it deploys the CAMARA QoD Provisioning API will set that CR field to `RUNNING`.

By setting these characteristics, you will rely on the value of `spec.camaraAPI.status` to set the service as `active`. The previous operator, when it deploys the CAMARA QoD Provisioning API, will set that CR field to `RUNNING`; until then it remains `NOT_RUNNING` and the resource stays pending. Optionally, you may also set `_CR_CHECKVAL_AVAILABLE = RUNNING` (bookkeeping only).

Then, you can proceed to create a CFS Service, which will incorporate the just created RFS Service. More information is available at: [Exposing Kubernetes Operators as a Service : Offering "Calculator as a Service" through OpenSlice](../../service_design/examples/calculator_crd_aas/calculator_crd_aas.md). To create the CFS Service characteristics, you may use the Service Specification available at `/QoDProvisioningAPI/OSLArtifacts/CAMARAaaS-QoD-Provisioning-API-CFS-Specification.json`. You may manually create the CFS Service, or you may onboard this Service Specification by making a POST request to `{{url}}/tmf-api/serviceCatalogManagement/v4/serviceSpecification`.

Pros, in a nutshell:

1. CRIDGE service allows OSL to:

    - Create and manage Custom Resources (CRs) using installed CRDs on a target Kubernetes cluster.

    - Facilitate complex orchestration scenarios by wrapping Kubernetes APIs as TMF APIs and models.

    - Handles connectivity to a Kubernetes cluster and manages the lifecycle of CRDs

    - Wraps the Kubernetes API, receives and provides resources towards other OSL services via the service bus

2. Enabling Loose Coupling and Orchestration

1. Enabling Loose Coupling and Orchestration

    - Language Flexibility: Developers can write CRDs in any language and expose them via the Kubernetes APIs. OSL will reuse these CRDs, enhancing flexibility and integration capabilities.

    - Familiar Deployment: Developers can create and deploy applications using familiar tools such as Helm charts, simplifying the process and reducing the learning curve.

3. Ecosystem Reusability

2. Ecosystem Reusability

    - CRIDGE capitalizes on the extensive Kubernetes ecosystem, particularly focusing on operators (CRDs).

    - Key repositories and hubs such as artifacthub.io and Operatorhub.io can be utilized for finding and deploying operators.

4. Service Catalog Exposure and Deployment

3. Service Catalog Exposure and Deployment

    OSL can expose CRs in service catalogs, facilitating their deployment in complex scenarios.

    These scenarios may include service bundles that involve multiple systems, such as RAN controllers or other Kubernetes clusters, providing a robust and versatile deployment framework.

    name: example_resource_1

```

In a nutchell:

 - Various OSL services use the Service Bus to communicate with CRIDGE.

 - CRIDGE converts requests towards Kubernetes API and vice-versa, facilitating the integration of custom resources with other OSL services.

 - CRDs are defined and managed through the K8s API. The example CRD is named myresource.example.com.

 - Deploying CRs in Namespaces: Custom resources defined by the CRD are created and managed within different namespaces in the Kubernetes cluster. Each namespace can have its own instances of the custom resources.

> The example CRD myresource.example.com allows the creation of custom resources of type Myresource.

> Instances of Myresource are created in various namespaces, each with unique names like example_resource_1.

> The example CRD myresource.example.com allows the creation of custom resources of type Myresource. Instances of Myresource are created in various namespaces, each with unique names like example_resource_1.

## Mupliple Clusters Management

- On CRIDGE start-up we try to register this cluster and context to OSL catalogs.

	- See registerKubernetesClientInOSLResource method which registers the KubernetesContextDefinition in Resource Inventory as a LogicalResource via  createOrUpdateResourceByNameCategoryVersion method

- After the creation (or update) of this cluster as a Resource in OSL we proceed to create SharedIndexInformers for CustomResourceDefinition objects

- In this way, CRIDGE is always aware of all CRDs and their CRs in the cluster, even if a CRD or CR is added/updated/deleted in the K8S cluster outside of OSL (CRIDGE)

- The SharedIndexInformer events notify CRIDGE, which is always aware of all CRDs and their CRs in the cluster, even if a CRD or CR is added/updated/deleted in the K8S cluster outside of OSL (CRIDGE)

	- NOTE: The ADD event is raised every time also we run CRIDGE. Therefore, on ADD we do the method to createORupdate resource specifications and resources

- On ADD event:

		- This label is added by OSOM during service orders to correlate K8S requested resources with resources in inventory

	- If the label exists, we update the resource by ID updateResourceById

	- Else a resource is created in catalog

	- A CR that is deleted from the cluster is reported with operationalState=DISABLE and administrativeState=SHUTDOWN

## Exposure of CRDs as Service Specifications 

## Service Orchestration and CRDs/CRs

OSOM checks the presence of attribute _CR_SPEC at the RFS to make a request for a CR deployment

- _CR_SPEC is a JSON or YAML string that is used for the request

	- It is similar to what one will do with e.g. a kubectl apply

	- There are tools to translate a yaml file to a json

> LCM rules can be used to change attributes of this yaml/json file, before sending this for orchestration

However, the following issue needs to be solved: **How to map the CR lifecycle that is defined in the CRD with the TMF resource Lifecycle?**

For this We introduced the following characteristics:  

- _CR_CHECK_FIELD

- CR_CHECKVAL_STANDBY

- _CR_CHECKVAL_ALARM 

- _CR_CHECKVAL_AVAILABLE

- _CR_CHECKVAL_RESERVED

- _CR_CHECKVAL_UNKNOWN

- _CR_CHECKVAL_SUSPENDED

OSOM sends to CRIDGE a message with the following information:

- currentContextCluster: current context of cluster 

- clusterMasterURL: current master url of the cluster 

- org.etsi.osl.serviceId: This is the related service id that the created resource has a reference 

- org.etsi.osl.resourceId: This is the related resource id that the created CR will wrap and reference.  

- org.etsi.osl.prefixName: we need to add a short prefix (default is cr) to various places. For example in K8s cannot start with a number 

- org.etsi.osl.serviceOrderId: the related service order id of this deployment request 

- org.etsi.osl.namespace: requested namespace name 

- org.etsi.osl.statusCheckFieldName: The name of the field that is needed to be monitored in order to monitor the status of the service and translate it to TMF resource statys (RESERVED AVAILABLE, etc) 

- org.etsi.osl.statusCheckValueStandby: The CR specific value (of the CheckFieldName) that needs to me mapped to the TMF resource state STANDBY (see org.etsi.osl.tmf.ri639.model.ResourceStatusType) 

- org.etsi.osl.statusCheckValueAlarm: The CR specific value (of the CheckFieldName) that needs to me mapped to the TMF resource state ALARMS (see org.etsi.osl.tmf.ri639.model.ResourceStatusType) 

- org.etsi.osl.statusCheckValueAvailable: The CR specific value (of the CheckFieldName) that needs to me mapped to the TMF resource state AVAILABLE (see org.etsi.osl.tmf.ri639.model.ResourceStatusType) 

- org.etsi.osl.statusCheckValueReserved: The CR specific value (of the CheckFieldName) that needs to me mapped to the TMF resource state RESERVED (see org.etsi.osl.tmf.ri639.model.ResourceStatusType) 

- org.etsi.osl.statusCheckValueUnknown: The CR specific value (of the CheckFieldName) that needs to me mapped to the TMF resource state UNKNOWN (see org.etsi.osl.tmf.ri639.model.ResourceStatusType) 

- org.etsi.osl.statusCheckValueSuspended: The CR specific value (of the CheckFieldName) that needs to me mapped to the TMF resource state SUSPENDED (see org.etsi.osl.tmf.ri639.model.ResourceStatusType) 

- Parameters:

	- aService reference to the service that the resource and the CR belongs to

	- resourceCR reference the equivalent resource in TMF repo of the target CR. One to one mapping

	- orderId related service order ID

	- startDate start date of the deployment (not used currently)

	- endDate end date of the deployment (not used currently)

	- _CR_SPEC the spec that is sent to CRIDGE (in json)

- Returns:

	- a string response from CRIDGE. It might return "OK" if everything is ok. "SEE OTHER" if there are multiple CRIDGEs then some other CRIDGE instance will handle the request for the equivalent cluster. Any other response is handled as error

- CRIDGE receives the message and creates according to the labels the necessary CR

- It monitors the created resource(s) in namespace (see the Sequence Diagram in previous images)

- It monitors and tries to figure out and map the Status of the CR to the TMF Status according to the provided org.etsi.osl.statusCheck* labels

- It sends to the message bus the current resource for creation or update to the TMF service inventory

OSOM requests a CR deployment when the RFS has a `_CR_SPEC` characteristic (see [Service Orchestration and CRDs/CRs](../service_design/kubernetes/exposing_kubernetes_resources.md#service-orchestration-and-crdscrs)).

How the CR lifecycle is mapped onto the TMF resource lifecycle through `_CR_CHECK_FIELD` and the `_CR_CHECKVAL_*` characteristics is described in [Mapping the CR lifecycle](../service_design/kubernetes/exposing_kubernetes_resources.md#mapping-the-cr-lifecycle-that-is-defined-in-the-crd-with-the-openslice-tmf-based-resource-lifecycle).

OSOM sends CRIDGE a message with the following headers:

- currentContextCluster: current context of the cluster

- clusterMasterURL: current master URL of the cluster

- org.etsi.osl.serviceId: the id of the service the created resource refers to

- org.etsi.osl.resourceId: the id of the resource that the created CR wraps (one-to-one)

- org.etsi.osl.prefixName: a short prefix (default `cr`) used in names, since K8s names cannot start with a number

- org.etsi.osl.serviceOrderId: the related service order id

- org.etsi.osl.namespace: the requested namespace name

- org.etsi.osl.statusCheckFieldName: the value of `_CR_CHECK_FIELD`

- org.etsi.osl.statusCheckValue{Standby|Alarm|Available|Reserved|Unknown|Suspended}: the values of the equivalent `_CR_CHECKVAL_*` resource status characteristics

- org.etsi.osl.{adminStateCheckValue*|operStateCheckValue*|usageStateCheckValue*|healthCheckValue*}: the values of the equivalent `_CR_CHECKVAL_ADMINSTATE_*`, `_CR_CHECKVAL_OPERSTATE_*`, `_CR_CHECKVAL_USAGESTATE_*` and `_CR_CHECKVAL_HEALTH_*` characteristics (e.g. `org.etsi.osl.healthCheckValueUp` carries `_CR_CHECKVAL_HEALTH_UP`)

The message body is the `_CR_SPEC` (in JSON). CRIDGE replies "OK" on success, or "SEE OTHER" if another CRIDGE instance handles the target cluster. Any other reply is treated as an error.

CRIDGE then:

- creates the CR, adding the headers as labels

- watches the created resource(s) in the namespace (see the Sequence Diagram in previous images)

- reads the value of the check field and resolves it, using the labels, into:

    - the resource `operationalState` / `administrativeState` / `usageState` (ITU-T X.731), which drive the service lifecycle

    - the `resourceStatus`, which is used for inventory / pool bookkeeping only

- sends the resource to the message bus, to be created or updated in the TMF resource inventory

## Deployment of a new CR based on a CRD