Update docs

# Deployment Guide

## Working scenarios

## Main flows

### `dual_oop`

### Dual OOP

Use this for real federation across two hosts.

Two hosts, real federation.

```bash

ansible-playbook playbooks/scenarios/dual_oop/deploy.yml --limit openop_2,openop_3

```

Validated kubernetes-adapter demo deployment:

```bash

ansible-playbook playbooks/scenarios/dual_oop/deploy.yml \

  --limit openop_2,openop_3 \

  -e oop_deployment_profile=kubernetes \

  -e srm_controller_image_tag=feature-srm-fm-integration-srmfix2-fb67c92

```

This deploys:

- OP1 on `op1_nodes`

- OP2 on `op2_nodes`

You can deploy one side only if needed:

`lite2edge` profile:

```bash

ansible-playbook playbooks/scenarios/dual_oop/deploy.yml --limit openop_3

ansible-playbook playbooks/scenarios/dual_oop/deploy.yml --limit openop_2

ansible-playbook ansible/playbooks/scenarios/dual_oop/deploy-lite2edge.yml \

  -i ansible/inventory/hosts.yml \

  --limit openop_2,openop_3

```

### `full_oop`

Use this for one complete Operator Platform on one host.

`kubernetes` profile:

```bash

ansible-playbook playbooks/scenarios/full_oop/deploy.yml --limit openop_3

ansible-playbook ansible/playbooks/scenarios/dual_oop/deploy-kubernetes.yml \

  -i ansible/inventory/hosts.yml \

  --limit openop_2,openop_3

```

## Deploy, redeploy, or undeploy one service

Use tool playbooks under `ansible/playbooks/tools/<service>/`.

Examples:

- deploy or reconcile Homer on both sides

Minimal profile switch on existing clusters:

```bash

ansible-playbook playbooks/tools/homer/deploy.yml -e target_hosts=openop_2,openop_3

```

- undeploy Homer on both sides of `dual_oop`

```bash

ansible-playbook playbooks/tools/homer/undeploy.yml -e target_hosts=openop_2,openop_3

ansible-playbook ansible/playbooks/scenarios/dual_oop/deploy-kubernetes.yml \

  -i ansible/inventory/hosts.yml \

  --limit openop_2,openop_3 \

  --tags srm,federation-manager

```

- undeploy Homer only on OP1

```bash

ansible-playbook playbooks/tools/homer/undeploy.yml -e target_hosts=openop_3

```

### Full OOP

- deploy or update SRM only on both sides, keeping custom image tag

One host, one complete platform.

```bash

ansible-playbook playbooks/tools/srm/deploy.yml \

  -e target_hosts=openop_2,openop_3 \

  -e srm_controller_image_tag=feature-srm-fm-integration-srmfix2-fb67c92

ansible-playbook ansible/playbooks/scenarios/full_oop/deploy.yml \

  -i ansible/inventory/hosts.yml \

  --limit openop_3

```

Notes:

- available tool playbooks include: `homer`, `srm`, `oeg`, `federation-manager`, `artefact-manager`, `zot`, `lite2edge`, `i2edge`

- use `-e target_hosts=openop_3` or `-e target_hosts=openop_2` for one side only

- use `-e target_hosts=openop_2,openop_3` to update both sides in `dual_oop`

- `/deploy.yml` is idempotent: if nothing changed, Ansible usually reports `ok` and does not restart or recreate service

- to apply changed templates, image tags, manifests, or vars, rerun same tool `/deploy.yml` command with those updated inputs

- tool playbooks use inventory host variables, including remote kubeconfig paths and filenames

- if you want to use scenario playbooks instead, `--tags <service>` works for deploy/reconcile, but dedicated tool `undeploy.yml` is clearer for removal

## Update one service

## Where kubeconfigs go

For remote deployments, playbooks fetch kubeconfigs locally to:

```text

~/kind-cluster-configs/<inventory-host>/

```

Use `--tags` on scenario playbooks when only a few services must change.

Examples:

- `~/kind-cluster-configs/openop_3/op1-kubeconfig.yaml`

- `~/kind-cluster-configs/openop_2/op2-kubeconfig.yaml`

Use them like this:

```bash

export KUBECONFIG=~/kind-cluster-configs/openop_3/op1-kubeconfig.yaml

kubectl get pods -A

```

## Access points

Useful service URLs:

ansible-playbook ansible/playbooks/scenarios/dual_oop/deploy-lite2edge.yml \

  -i ansible/inventory/hosts.yml \

  --limit openop_2,openop_3 \

  --tags srm,oeg,federation-manager

- OEG northbound: `http://<host-ip>:32263/oeg/1.0.0`

- Federation Manager: `http://<host-ip>:30989`

- Homer: `http://<host-ip>:30088`

- Zot: `http://<host-ip>:30050`

- Prometheus: `http://<host-ip>:30090`

- Grafana: `http://<host-ip>:30091`

ansible-playbook ansible/playbooks/scenarios/dual_oop/deploy-lite2edge.yml \

  -i ansible/inventory/hosts.yml \

  --limit openop_2,openop_3 \

  --tags lite2edge

```

## End-to-end smoke test

## Smoke tests

After `dual_oop` is green, run smoke from `local-docker-deployment/` in repository root.

Run from repository root.

Validated full lifecycle smoke for real two-host kubernetes-adapter demo:

Container smoke:

```bash

APP_ID=18909090-9090-4909-8909-189090909090 \

APP_NAME=federatednginxoci \

KEYCLOAK_TOKEN_URL=http://192.168.123.155:30082/realms/federation/protocol/openid-connect/token \

CLIENT_ID=originating-op-1 \

CLIENT_SECRET=dd7vNwFqjNpYwaghlEwMbw10g0klWDHb \

OEG_BASE_URL=http://192.168.123.155:32263/oeg/1.0.0 \

REMOTE_OEG_BASE_URL=http://192.168.123.178:32263/oeg/1.0.0 \

LOCAL_ZONE_ID=47056b13-494d-4e68-a7e7-88a783d69b1d \

LOCAL_ZONE_NAME=op1-control-plane \

FEDERATED_ZONE_ID=5460f101-e44c-4dae-b41b-cb7752aafced \

APP_PACKAGE_TYPE=CONTAINER \

APP_IMAGE_PATH=nginx:latest \

REMOTE_SRM_BASE_URL=http://192.168.123.178:32415/srm/1.0.0 \

KEYCLOAK_TOKEN_URL=http://192.168.123.155:30082/realms/federation/protocol/openid-connect/token \

LOCAL_PROVIDER="Local Operator" \

FEDERATED_PROVIDER="Remote Operator" \

PARTNER_STATUS_LINK=http://192.168.123.155:32263/oeg/1.0.0/partner \

ZONE_NOTIF_LINK=http://192.168.123.155:32263/oeg/1.0.0/edge-cloud-zones \

REUSE_FEDERATION_CONTEXT=false \

CLEANUP_FEDERATION=true \

bash ../local-docker-deployment/smoke-test.sh

```

Deploy-only variant for manual API E2E testing:

```bash

APP_ID=18707070-7070-4707-8707-187070707070 \

APP_NAME=federatednginxoci \

KEYCLOAK_TOKEN_URL=http://192.168.123.155:30082/realms/federation/protocol/openid-connect/token \

CLIENT_ID=originating-op-1 \

CLIENT_SECRET=dd7vNwFqjNpYwaghlEwMbw10g0klWDHb \

OEG_BASE_URL=http://192.168.123.155:32263/oeg/1.0.0 \

REMOTE_OEG_BASE_URL=http://192.168.123.178:32263/oeg/1.0.0 \

LOCAL_ZONE_ID=47056b13-494d-4e68-a7e7-88a783d69b1d \

LOCAL_ZONE_NAME=op1-control-plane \

FEDERATED_ZONE_ID=5460f101-e44c-4dae-b41b-cb7752aafced \

CLEAN_EXISTING_INSTANCES=true \

APP_PACKAGE_TYPE=CONTAINER \

APP_REPO_TYPE=PUBLICREPO \

APP_IMAGE_PATH=nginx:latest \

REUSE_FEDERATION_CONTEXT=false \

CLEANUP_FEDERATION=false \

bash ../local-docker-deployment/smoke-test-deploy.sh

APP_PROVIDER=Local_Oper \

SMOKE_STATE_FILE=/tmp/opencode/ansible-smoke.state \

bash local-docker-deployment/smoke-test.sh

```

Cleanup after deploy-only demo:

HELM smoke:

```bash

APP_ID=18707070-7070-4707-8707-187070707070 \

APP_NAME=federatednginxoci \

KEYCLOAK_TOKEN_URL=http://192.168.123.155:30082/realms/federation/protocol/openid-connect/token \

CLIENT_ID=originating-op-1 \

CLIENT_SECRET=dd7vNwFqjNpYwaghlEwMbw10g0klWDHb \

OEG_BASE_URL=http://192.168.123.155:32263/oeg/1.0.0 \

REMOTE_OEG_BASE_URL=http://192.168.123.178:32263/oeg/1.0.0 \

LOCAL_ZONE_ID=47056b13-494d-4e68-a7e7-88a783d69b1d \

LOCAL_ZONE_NAME=op1-control-plane \

FEDERATED_ZONE_ID=5460f101-e44c-4dae-b41b-cb7752aafced \

APP_PACKAGE_TYPE=CONTAINER \

APP_IMAGE_PATH=nginx:latest \

REMOTE_SRM_BASE_URL=http://192.168.123.178:32415/srm/1.0.0 \

KEYCLOAK_TOKEN_URL=http://192.168.123.155:30082/realms/federation/protocol/openid-connect/token \

LOCAL_PROVIDER="Local Operator" \

FEDERATED_PROVIDER="Remote Operator" \

PARTNER_STATUS_LINK=http://192.168.123.155:32263/oeg/1.0.0/partner \

ZONE_NOTIF_LINK=http://192.168.123.155:32263/oeg/1.0.0/edge-cloud-zones \

REUSE_FEDERATION_CONTEXT=false \

CLEANUP_FEDERATION=true \

bash ../local-docker-deployment/smoke-test-undeploy.sh

CLEAN_EXISTING_INSTANCES=true \

APP_PACKAGE_TYPE=HELM \

APP_REPO_TYPE=PUBLICREPO \

APP_IMAGE_PATH=registry-1.docker.io/bitnamicharts/nginx:23.0.2 \

APP_PROVIDER=Local_Oper \

SMOKE_STATE_FILE=/tmp/opencode/ansible-smoke-helm.state \

bash local-docker-deployment/smoke-test.sh

```

Manual API E2E testing sequence:

1. Local deployment:

   use `manual-api-e2e-testing.md` section `5` with OP1 OEG URL `http://192.168.123.155:32263/oeg/1.0.0`, local zone `47056b13-494d-4e68-a7e7-88a783d69b1d`, `APP_PACKAGE_TYPE=CONTAINER`, `APP_IMAGE_PATH=nginx:latest`

2. Federation enablement:

   use `manual-api-e2e-testing.md` section `6` against OP1 OEG

3. Federated deployment:

   use `manual-api-e2e-testing.md` section `7` with remote zone `5460f101-e44c-4dae-b41b-cb7752aafced`

4. Verification:

   `curl -sS "$OEG_BASE_URL/appinstances?appId=<APP_ID>"`

5. Optional partner-side proof:

   `curl -sS "http://192.168.123.178:32415/srm/1.0.0/internal/fm/deployments/<APP_ID>/instances/<REMOTE_INSTANCE_ID>/zones/5460f101-e44c-4dae-b41b-cb7752aafced"`

Full step-by-step manual `curl` walkthrough:

- [Manual API E2E Testing](manual-api-e2e-testing.md)

Notes:

- use OEG northbound URL on port `32263`, not Federation Manager on `30989`

- for kubernetes-adapter demo, use `APP_PACKAGE_TYPE=CONTAINER` and `APP_IMAGE_PATH=nginx:latest`

- current smoke script accepts both `ready` and `READY` when checking instance readiness

- script deploys and then cleans up app instances in one run

Expected success signals:

- local app instance reaches `ready`

- federated app instance reaches `ready` or `READY`

- cleanup leaves `Remaining matching app instances: []`

- use `REMOTE_SRM_BASE_URL`, not `REMOTE_OEG_BASE_URL`, before federation exists

- use isolated `SMOKE_STATE_FILE` values for ansible smoke

- validated matrix:

  - `lite2edge + CONTAINER + local/federated`

  - `lite2edge + HELM + local/federated`

  - `kubernetes + CONTAINER + local/federated`

  - `kubernetes + HELM` not supported

## Undeploy

Quick removal deletes clusters directly:

```bash

ansible-playbook playbooks/scenarios/dual_oop/quick_undeploy.yml --limit openop_2,openop_3

ansible-playbook playbooks/scenarios/full_oop/quick_undeploy.yml --limit openop_3

```

Graceful removal undeploys components first:

```bash

ansible-playbook playbooks/scenarios/dual_oop/graceful_undeploy.yml --limit openop_2,openop_3

ansible-playbook playbooks/scenarios/full_oop/graceful_undeploy.yml --limit openop_3

```

## Troubleshooting

### Port already in use

If deployment fails before Kind cluster creation, old clusters or containers still own NodePort mappings. Remove stale Kind clusters on target hosts, then retry.

### SRM or another component stays in `FAILED - RETRYING`

Read pod status with fetched kubeconfig:

Quick removal:

```bash

export KUBECONFIG=~/kind-cluster-configs/openop_3/op1-kubeconfig.yaml

kubectl get pods -A

kubectl describe pod -n oop <pod-name>

ansible-playbook ansible/playbooks/scenarios/dual_oop/quick_undeploy.yml \

  -i ansible/inventory/hosts.yml \

  --limit openop_2,openop_3

```

### Verify cluster health

Graceful removal:

```bash

kubectl get nodes

kubectl get pods -A

kubectl get svc -A

ansible-playbook ansible/playbooks/scenarios/dual_oop/graceful_undeploy.yml \

  -i ansible/inventory/hosts.yml \

  --limit openop_2,openop_3

```

- Linux machine with Python 3

- SSH access to target hosts

- enough resources on each VM for Kind and platform components

- enough disk and RAM on each VM for Kind and platform services

## 1. Prepare the Ansible environment

## Prepare Ansible

```bash

cd ansible

python3 -m venv venv

source venv/bin/activate

pip install -r requirements.txt

ansible-galaxy collection install -r requirements.yml

cd /home/sergio/i2cat/OperatorPlatform/OOP/OP_Automation

python3 -m venv ansible/venv

source ansible/venv/bin/activate

pip install -r ansible/requirements.txt

ansible-galaxy collection install -r ansible/requirements.yml

```

## 2. Configure the inventory

## Configure inventory

Edit `ansible/inventory/hosts.yml`.

Important groups:

- `op1_nodes`: first Operator Platform in `dual_oop`

- `op2_nodes`: second Operator Platform in `dual_oop`

- `op1_nodes`: first Operator Platform

- `op2_nodes`: second Operator Platform

- `k8s_clusters`: shared prerequisite group

Minimal remote-host example:

Test SSH and Python:

```yaml

all:

  hosts:

    openop_3:

      ansible_host: 192.168.123.155

      ansible_connection: ssh

      ansible_user: ubuntu

      ansible_ssh_private_key_file: ~/.ssh/your_key

      ansible_python_interpreter: /usr/bin/python3

      deployment_mode: remote

```bash

ansible all -i ansible/inventory/hosts.yml -m ping

```

## 3. Create `ansible/secrets.yml`

## Create secrets file

Create `ansible/secrets.yml` with credentials needed for private images.

Create `ansible/secrets.yml` with GitLab registry credentials.

Example:

```yaml

gitlab_token: "your-token"

lite2edge_registry_username: "<gitlab-user>"

gitlab_token: "<gitlab-token>"

```

Notes:

## Deploy

- file is gitignored

- `dual_oop` and `full_oop` load it automatically

## 4. Test connectivity

`lite2edge` profile on two hosts:

```bash

ansible all -i inventory/hosts.yml -m ping

ansible-playbook ansible/playbooks/scenarios/dual_oop/deploy-lite2edge.yml \

  -i ansible/inventory/hosts.yml \

  --limit openop_2,openop_3

```

## 5. Run a deployment

Two-host federation setup:

`kubernetes` profile on two hosts:

```bash

ansible-playbook playbooks/scenarios/dual_oop/deploy.yml --limit openop_2,openop_3

ansible-playbook ansible/playbooks/scenarios/dual_oop/deploy-kubernetes.yml \

  -i ansible/inventory/hosts.yml \

  --limit openop_2,openop_3

```

Validated kubernetes-adapter demo deployment:

Switch an existing dual deployment from `lite2edge` to `kubernetes` with minimal touch:

```bash

ansible-playbook playbooks/scenarios/dual_oop/deploy.yml \

ansible-playbook ansible/playbooks/scenarios/dual_oop/deploy-kubernetes.yml \

  -i ansible/inventory/hosts.yml \

  --limit openop_2,openop_3 \

  -e oop_deployment_profile=kubernetes \

  -e srm_controller_image_tag=feature-srm-fm-integration-srmfix2-fb67c92

```

Current known-good image tags in workspace:

- SRM: `feature-srm-fm-integration-srmfix2-fb67c92`

- OEG: `feature-srm-fm-integration-a6aea77`

- FM: `feature-srm-fm-integration-wip-f74aae0`

Single-host complete platform:

```bash

ansible-playbook playbooks/scenarios/full_oop/deploy.yml --limit openop_3

  --tags srm,federation-manager

```

Apply one service only:

## Verify

```bash

ansible-playbook playbooks/tools/homer/deploy.yml -e target_hosts=openop_2,openop_3

```

Undeploy one service only:

```bash

ansible-playbook playbooks/tools/homer/undeploy.yml -e target_hosts=openop_2,openop_3

```

If nothing changed, rerunning tool `/deploy.yml` is usually a no-op.

More single-service examples: `docs/deployment.md`.

## 6. Verify the result

For remote deployments, kubeconfigs are fetched locally to:

Remote kubeconfigs are fetched locally to:

```text

~/kind-cluster-configs/<inventory-host>/

export KUBECONFIG=~/kind-cluster-configs/openop_3/op1-kubeconfig.yaml

kubectl get nodes

kubectl get pods -A

```

Useful demo endpoints after `dual_oop` deploy:

- OP1 OEG: `http://192.168.123.155:32263/oeg/1.0.0`

- OP2 OEG: `http://192.168.123.178:32263/oeg/1.0.0`

- OP1 SRM: `http://192.168.123.155:32415/srm/1.0.0`

- OP2 SRM: `http://192.168.123.178:32415/srm/1.0.0`

- Keycloak token endpoint: `http://192.168.123.155:30082/realms/federation/protocol/openid-connect/token`

Useful demo zone ids:

- OP1 local zone: `47056b13-494d-4e68-a7e7-88a783d69b1d`

- OP2 local zone: `5460f101-e44c-4dae-b41b-cb7752aafced`

export KUBECONFIG=~/kind-cluster-configs/openop_2/op2-kubeconfig.yaml

kubectl get nodes

kubectl get pods -A

```

## 7. Undeploy

## Undeploy

Quick removal:

```bash

ansible-playbook playbooks/scenarios/dual_oop/quick_undeploy.yml --limit openop_2,openop_3

ansible-playbook playbooks/scenarios/full_oop/quick_undeploy.yml --limit openop_3

ansible-playbook ansible/playbooks/scenarios/dual_oop/quick_undeploy.yml \

  -i ansible/inventory/hosts.yml \

  --limit openop_2,openop_3

```

Graceful removal:

```bash

ansible-playbook playbooks/scenarios/dual_oop/graceful_undeploy.yml --limit openop_2,openop_3

ansible-playbook playbooks/scenarios/full_oop/graceful_undeploy.yml --limit openop_3

ansible-playbook ansible/playbooks/scenarios/dual_oop/graceful_undeploy.yml \

  -i ansible/inventory/hosts.yml \

  --limit openop_2,openop_3

```

# Operator Platform Automation

Docs for the currently used Ansible deployment flows.

Brief docs for current Ansible flows.

## Start here

## Read first

1. Read [Getting Started](getting-started.md).

2. Read [Deployment Guide](deployment.md).

3. For manual API E2E testing, use Deployment Guide section `End-to-end smoke test`.

4. For step-by-step manual `curl` flow, read [Manual API E2E Testing](manual-api-e2e-testing.md).

1. [Getting Started](getting-started.md)

2. [Deployment Guide](deployment.md)

3. [Manual API E2E Testing](manual-api-e2e-testing.md)

## Working paths

## Current validated scope

- `dual_oop`: two hosts, real federation between two Operator Platforms

- `full_oop`: one host, one complete Operator Platform

- `dual_oop` on `openop_3` + `openop_2`

- `lite2edge + CONTAINER + local/federated`

- `lite2edge + HELM + local/federated`

- `kubernetes + CONTAINER + local/federated`

- `kubernetes + HELM` not supported

## Useful defaults

## Release images

- OEG northbound: `http://<host-ip>:32263/oeg/1.0.0`

- Federation Manager: `http://<host-ip>:30989`

- SRM: `1.1`

- OEG: `1.1`

- FM: `1.1`

## Useful endpoints

- OEG: `http://<host-ip>:32263/oeg/1.0.0`

- SRM: `http://<host-ip>:32415/srm/1.0.0`

- Federation Manager: `http://<host-ip>:30989`

- Homer: `http://<host-ip>:30088`

- Zot: `http://<host-ip>:30050`

- Kubeconfigs fetched locally: `~/kind-cluster-configs/<inventory-host>/`

## Notes

- Run playbooks from `ansible/`.

- Run playbooks from `OOP/OP_Automation/`.

- Use `-i ansible/inventory/hosts.yml`.

- `ansible/secrets.yml` is local-only and gitignored.

- Remote smoke tests talk to OEG northbound (`32263`), not Federation Manager (`30989`).

- Current validated k8s-adapter demo uses SRM tag `feature-srm-fm-integration-srmfix2-fb67c92`.

- For ansible smoke, use `REMOTE_SRM_BASE_URL`, not `REMOTE_OEG_BASE_URL`, before federation exists.