update docs

Manual API E2E testing sequence:

1. Local deployment:

   use `MANUAL_E2E_WALKTHROUGH.md` section `6` with OP1 OEG URL `http://192.168.123.155:32263/oeg/1.0.0`, local zone `47056b13-494d-4e68-a7e7-88a783d69b1d`, `APP_PACKAGE_TYPE=CONTAINER`, `APP_IMAGE_PATH=nginx:latest`

   use `manual-api-e2e-testing.md` section `5` with OP1 OEG URL `http://192.168.123.155:32263/oeg/1.0.0`, local zone `47056b13-494d-4e68-a7e7-88a783d69b1d`, `APP_PACKAGE_TYPE=CONTAINER`, `APP_IMAGE_PATH=nginx:latest`

2. Federation enablement:

   use section `7.1` to `7.4` against OP1 OEG

   use `manual-api-e2e-testing.md` section `6` against OP1 OEG

3. Federated deployment:

   use section `7.6` with remote zone `5460f101-e44c-4dae-b41b-cb7752aafced`

   use `manual-api-e2e-testing.md` section `7` with remote zone `5460f101-e44c-4dae-b41b-cb7752aafced`

4. Verification:

   `curl -sS "$OEG_BASE_URL/appinstances?appId=<APP_ID>"`

5. Optional partner-side proof:

   `curl -sS "http://192.168.123.178:32415/srm/1.0.0/internal/fm/deployments/<APP_ID>/instances/<REMOTE_INSTANCE_ID>/zones/5460f101-e44c-4dae-b41b-cb7752aafced"`

Full step-by-step manual `curl` walkthrough:

- [Manual API E2E Testing](manual-api-e2e-testing.md)

Notes:

- use OEG northbound URL on port `32263`, not Federation Manager on `30989`

1. Read [Getting Started](getting-started.md).

2. Read [Deployment Guide](deployment.md).

3. For manual API E2E testing, use Deployment Guide section `End-to-end smoke test`.

4. For step-by-step manual `curl` flow, read [Manual API E2E Testing](manual-api-e2e-testing.md).

## Working paths

# Manual API E2E Testing

This guide explains how to:

- deploy real two-host `dual_oop` with Ansible

- export and use fetched kubeconfigs

- inspect both Kubernetes clusters

- manually drive OEG northbound APIs for:

  - local deployment

  - federation enablement

  - federated deployment

  - cleanup

This is prod-ready counterpart to `local-docker-deployment/MANUAL_E2E_WALKTHROUGH.md`.

Same OEG API sequence. Different deployment path, URLs, zone ids, kubeconfigs.

## Validated Scope

This walkthrough is validated for:

1. `dual_oop` two-host deployment

2. local SRM using `kubernetes` adapter

3. OEG northbound driven manually with `curl`

4. app payload using:

   - `APP_PACKAGE_TYPE=CONTAINER`

   - `APP_IMAGE_PATH=nginx:latest`

Current known-good image tags:

- SRM: `feature-srm-fm-integration-srmfix2-fb67c92`

- OEG: `feature-srm-fm-integration-a6aea77`

- FM: `feature-srm-fm-integration-wip-f74aae0`

## Prerequisites

- inventory already configured in `ansible/inventory/hosts.yml`

- `ansible/secrets.yml` present with registry token

- host can SSH to both target VMs

- `curl`, `kubectl`, Python 3 available locally

Work from:

```bash

cd /home/sergio/i2cat/OperatorPlatform/OOP/OP_Automation/ansible

```

## 1. Deploy Two-Host Platform

Validated deployment command:

```bash

ansible-playbook playbooks/scenarios/dual_oop/deploy.yml \

  --limit openop_2,openop_3 \

  -e oop_deployment_profile=kubernetes \

  -e srm_controller_image_tag=feature-srm-fm-integration-srmfix2-fb67c92

```

Expected host mapping in current environment:

- OP1: `openop_3` at `192.168.123.155`

- OP2: `openop_2` at `192.168.123.178`

## 2. Export Kubeconfigs

Fetched kubeconfigs live under:

```text

~/kind-cluster-configs/<inventory-host>/

```

Current files:

- OP1: `~/kind-cluster-configs/openop_3/op1-kubeconfig.yaml`

- OP2: `~/kind-cluster-configs/openop_2/op2-kubeconfig.yaml`

Quick checks:

```bash

export KUBECONFIG=~/kind-cluster-configs/openop_3/op1-kubeconfig.yaml

kubectl get nodes

kubectl get pods -A

export KUBECONFIG=~/kind-cluster-configs/openop_2/op2-kubeconfig.yaml

kubectl get nodes

kubectl get pods -A

```

## 3. Common Variables

Set variables once before manual flow:

```bash

export OEG_BASE_URL="http://192.168.123.155:32263/oeg/1.0.0"

export REMOTE_OEG_BASE_URL="http://192.168.123.178:32263/oeg/1.0.0"

export KEYCLOAK_TOKEN_URL="http://192.168.123.155:30082/realms/federation/protocol/openid-connect/token"

export APP_ID="18909090-9090-4909-8909-189090909090"

export APP_NAME="federatednginxoci"

export APP_PROVIDER="Local_Oper"

export APP_VERSION="1"

export APP_PACKAGE_TYPE="CONTAINER"

export APP_IMAGE_PATH="nginx:latest"

export COMPONENT_NAME="frontend"

export LOCAL_PROVIDER="Local Operator"

export LOCAL_ZONE_ID="47056b13-494d-4e68-a7e7-88a783d69b1d"

export LOCAL_ZONE_NAME="op1-control-plane"

export LOCAL_ZONE_STATUS="active"

export FEDERATED_PROVIDER="Remote Operator"

export FEDERATED_ZONE_ID="5460f101-e44c-4dae-b41b-cb7752aafced"

```

Notes:

- use unique `APP_ID` per fresh run

- use OEG northbound `32263`, not Federation Manager `30989`

- for current k8s path, use `nginx:latest`, not Helm nginx chart path

## 4. Inspect Available Zones in OEG

Before federation, local OEG should show local zone:

```bash

curl -sS "$OEG_BASE_URL/edge-cloud-zones"

```

You should see at least:

- `Local Operator` / `47056b13-494d-4e68-a7e7-88a783d69b1d`

Remote zone appears only after federation context exists and zone is subscribed.

## 5. Manual Local Deployment Flow

### 5.1 Onboard app in OEG

```bash

curl -sS -X POST "$OEG_BASE_URL/apps" \

  -H 'Content-Type: application/json' \

  --data @- <<EOF

{

  "appId": "$APP_ID",

  "name": "$APP_NAME",

  "appProvider": "$APP_PROVIDER",

  "version": "$APP_VERSION",

  "packageType": "$APP_PACKAGE_TYPE",

  "appRepo": {

    "type": "PUBLICREPO",

    "imagePath": "$APP_IMAGE_PATH"

  },

  "requiredResources": {

    "infraKind": "container",

    "numCPU": "100m",

    "memory": 512

  },

  "componentSpec": [{

    "componentName": "$COMPONENT_NAME",

    "networkInterfaces": [{

      "interfaceId": "eth0",

      "protocol": "TCP",

      "port": 80,

      "visibilityType": "VISIBILITY_EXTERNAL"

    }]

  }]

}

EOF

```

Inspect app:

```bash

curl -sS "$OEG_BASE_URL/apps/$APP_ID"

```

### 5.2 Deploy app to local operator

```bash

LOCAL_DEPLOY_RESPONSE="$({

  curl -sS -X POST "$OEG_BASE_URL/appinstances" \

    -H 'Content-Type: application/json' \

    --data @- <<EOF

{

  "appId": "$APP_ID",

  "appZones": [{

    "EdgeCloudZone": {

      "edgeCloudZoneId": "$LOCAL_ZONE_ID",

      "edgeCloudZoneName": "$LOCAL_ZONE_NAME",

      "edgeCloudProvider": "$LOCAL_PROVIDER",

      "edgeCloudZoneStatus": "$LOCAL_ZONE_STATUS",

      "edgeCloudRegion": "unknown"

    }

  }]

}

EOF

} )"

printf '%s\n' "$LOCAL_DEPLOY_RESPONSE"

```

Extract local instance id:

```bash

export LOCAL_INSTANCE_ID="$(printf '%s' "$LOCAL_DEPLOY_RESPONSE" | sed -n 's/.*"appInstanceId":"\([^"]*\)".*/\1/p')"

printf 'LOCAL_INSTANCE_ID=%s\n' "$LOCAL_INSTANCE_ID"

```

Check instance list:

```bash

curl -sS "$OEG_BASE_URL/appinstances?appId=$APP_ID"

```

Expected local status:

- `ready`

Optional OP1 cluster proof:

```bash

export KUBECONFIG=~/kind-cluster-configs/openop_3/op1-kubeconfig.yaml

kubectl get deploy,pod,svc -n oop | grep "$APP_NAME"

```

## 6. Manual Federation Enablement Flow

### 6.1 Wait for Keycloak token endpoint

```bash

until curl -fsS -X POST "$KEYCLOAK_TOKEN_URL" \

  -H 'Content-Type: application/x-www-form-urlencoded' \

  --data 'client_id=originating-op-1&client_secret=dd7vNwFqjNpYwaghlEwMbw10g0klWDHb&grant_type=client_credentials' \

  >/dev/null; do

  sleep 2

done

```

### 6.2 Create federation context

```bash

FED_CREATE_RESPONSE="$({

  curl -sS -X POST "$OEG_BASE_URL/partner" \

    -H 'Content-Type: application/json' \

    --data @- <<EOF

{

  "origOPFederationId": "oop-local",

  "initialDate": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",

  "partnerStatusLink": "http://oeg.oop.svc.cluster.local/oeg/1.0.0/partner/notifications"

}

EOF

} )"

printf '%s\n' "$FED_CREATE_RESPONSE"

```

Extract federation id:

```bash

export FEDERATION_CONTEXT_ID="$(printf '%s' "$FED_CREATE_RESPONSE" | sed -n 's/.*"federationContextId"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p')"

printf 'FEDERATION_CONTEXT_ID=%s\n' "$FEDERATION_CONTEXT_ID"

```

If needed, current federation id:

```bash

curl -sS "$OEG_BASE_URL/fed-context-id"

```

### 6.3 Subscribe remote zone

```bash

curl -sS -X POST "$OEG_BASE_URL/$FEDERATION_CONTEXT_ID/zones" \

  -H 'Content-Type: application/json' \

  --data @- <<EOF

{

  "acceptedAvailabilityZones": ["$FEDERATED_ZONE_ID"],

  "availZoneNotifLink": "http://oeg.oop.svc.cluster.local/oeg/1.0.0/edge-cloud-zones/notifications"

}

EOF

```

Notes:

- `409` acceptable if zone already subscribed

- current valid remote zone id is `5460f101-e44c-4dae-b41b-cb7752aafced`

### 6.4 Wait for remote zone in OEG

```bash

curl -sS "$OEG_BASE_URL/edge-cloud-zones"

```

Repeat until you see:

- `edgeCloudProvider: "Remote Operator"`

- `edgeCloudZoneId: "5460f101-e44c-4dae-b41b-cb7752aafced"`

Optional federation detail check:

```bash

curl -sS "$OEG_BASE_URL/$FEDERATION_CONTEXT_ID/partner"

```

## 7. Manual Federated Deployment Flow

### 7.1 Deploy same app to remote operator

```bash

REMOTE_DEPLOY_RESPONSE="$({

  curl -sS -X POST "$OEG_BASE_URL/appinstances" \

    -H 'Content-Type: application/json' \

    --data @- <<EOF

{

  "appId": "$APP_ID",

  "appZones": [{

    "EdgeCloudZone": {

      "edgeCloudZoneId": "$FEDERATED_ZONE_ID",

      "edgeCloudZoneName": "unknown",

      "edgeCloudProvider": "$FEDERATED_PROVIDER",

      "edgeCloudZoneStatus": "unknown",

      "edgeCloudRegion": "unknown"

    }

  }]

}

EOF

} )"

printf '%s\n' "$REMOTE_DEPLOY_RESPONSE"

```

Extract remote instance id:

```bash

export REMOTE_INSTANCE_ID="$(printf '%s' "$REMOTE_DEPLOY_RESPONSE" | sed -n 's/.*"appInstIdentifier":"\([^"]*\)".*/\1/p')"

printf 'REMOTE_INSTANCE_ID=%s\n' "$REMOTE_INSTANCE_ID"

```

### 7.2 Check app instances

```bash

curl -sS "$OEG_BASE_URL/appinstances?appId=$APP_ID"

```

Expected result:

- local instance in `Local Operator`

- remote instance in `Remote Operator`

- local status `ready`

- remote status `READY` or `ready`

### 7.3 Optional partner-side proof

FM-facing SRM details endpoint on OP2:

```bash

curl -sS "http://192.168.123.178:32415/srm/1.0.0/internal/fm/deployments/$APP_ID/instances/$REMOTE_INSTANCE_ID/zones/$FEDERATED_ZONE_ID"

```

Expected key signal:

- `"appInstanceState": "ready"`

Optional OP2 cluster proof:

```bash

export KUBECONFIG=~/kind-cluster-configs/openop_2/op2-kubeconfig.yaml

kubectl get deploy,pod,svc -n oop | grep "$APP_NAME"

```

## 8. Manual Full E2E Order

1. Deploy `dual_oop` with kubernetes profile.

2. Export kubeconfigs.

3. Set common variables.

4. Onboard app in OP1 OEG.

5. Deploy app to OP1 local zone.

6. Wait for Keycloak.

7. Create federation context.

8. Subscribe OP2 zone.

9. Wait for remote zone to appear in OP1 OEG.

10. Deploy app to remote zone through OP1 OEG.

11. Verify `GET /appinstances?appId=...` returns local + remote instances.

12. Verify optional OP2 SRM FM-facing endpoint.

13. Clean up instances, app, zone subscription, federation.

## 9. Manual Cleanup

Cleanup order matters.

### 9.1 Delete app instances

List:

```bash

curl -sS "$OEG_BASE_URL/appinstances?appId=$APP_ID"

```

Delete local instance if present:

```bash

curl -sS -X DELETE "$OEG_BASE_URL/appinstances/$LOCAL_INSTANCE_ID"

```

Delete remote instance if present:

```bash

curl -sS -X DELETE "$OEG_BASE_URL/appinstances/$REMOTE_INSTANCE_ID"

```

Verify:

```bash

curl -sS "$OEG_BASE_URL/appinstances?appId=$APP_ID"

# Expected: []

```

### 9.2 Delete onboarded app

```bash

curl -sS -X DELETE "$OEG_BASE_URL/apps/$APP_ID"

```

### 9.3 Unsubscribe remote zone

```bash

curl -sS -X DELETE "$OEG_BASE_URL/$FEDERATION_CONTEXT_ID/zones/$FEDERATED_ZONE_ID"

```

### 9.4 Delete federation context

```bash

curl -sS -X DELETE "$OEG_BASE_URL/$FEDERATION_CONTEXT_ID/partner"

```

Verify cleanup:

```bash

curl -sS "$OEG_BASE_URL/appinstances?appId=$APP_ID"

curl -sS "http://192.168.123.178:32415/srm/1.0.0/deployedServiceFunction"

```

Expected:

- app instances `[]`

- no leftover OP2 deployed instance for test app

## 10. Useful Debugging Commands

OEG checks:

```bash

curl -sS "$OEG_BASE_URL/edge-cloud-zones"

curl -sS "$OEG_BASE_URL/apps/$APP_ID"

curl -sS "$OEG_BASE_URL/appinstances?appId=$APP_ID"

```

SRM checks:

```bash

curl -sS "http://192.168.123.155:32415/srm/1.0.0/node"

curl -sS "http://192.168.123.178:32415/srm/1.0.0/node"

curl -sS "http://192.168.123.178:32415/srm/1.0.0/deployedServiceFunction"

```

Cluster checks:

```bash

export KUBECONFIG=~/kind-cluster-configs/openop_3/op1-kubeconfig.yaml

kubectl get pods -A

kubectl get events -A --sort-by=.metadata.creationTimestamp

export KUBECONFIG=~/kind-cluster-configs/openop_2/op2-kubeconfig.yaml

kubectl get pods -A

kubectl get events -A --sort-by=.metadata.creationTimestamp

```

Ansible ad-hoc log pulls:

```bash

ansible openop_3 -i inventory/hosts.yml -m shell -a "kubectl logs -n oop deploy/oegcontroller --since=5m"

ansible openop_2 -i inventory/hosts.yml -m shell -a "kubectl logs -n federation-manager deploy/federation-manager --since=5m"

ansible openop_2 -i inventory/hosts.yml -m shell -a "kubectl logs -n oop deploy/srmcontroller --since=5m"

```

Common failure pattern:

- remote deploy `409 CONFLICT` often means stale OP2 `federatednginxoci` still exists

- clear old OP2 deployed instance, retry remote deploy

## 11. Relation to Scripted Checks

This walkthrough is manual equivalent of validated scripted checks in `local-docker-deployment/`.

Relevant command:

```bash

APP_ID=<new-uuid> \

APP_NAME=federatednginxoci \

KEYCLOAK_TOKEN_URL=http://192.168.123.155:30082/realms/federation/protocol/openid-connect/token \

CLIENT_ID=originating-op-1 \

CLIENT_SECRET=dd7vNwFqjNpYwaghlEwMbw10g0klWDHb \

OEG_BASE_URL=http://192.168.123.155:32263/oeg/1.0.0 \

REMOTE_OEG_BASE_URL=http://192.168.123.178:32263/oeg/1.0.0 \

LOCAL_ZONE_ID=47056b13-494d-4e68-a7e7-88a783d69b1d \

LOCAL_ZONE_NAME=op1-control-plane \

FEDERATED_ZONE_ID=5460f101-e44c-4dae-b41b-cb7752aafced \

APP_PACKAGE_TYPE=CONTAINER \

APP_IMAGE_PATH=nginx:latest \

REUSE_FEDERATION_CONTEXT=false \

CLEANUP_FEDERATION=true \

bash /home/sergio/i2cat/OperatorPlatform/local-docker-deployment/smoke-test.sh

```