Commit 7cc17c92 authored by Sergio Gimenez's avatar Sergio Gimenez
Browse files

docs(r2): describe what each Release 2 playbook sets up

playbooks/scenarios/r2_existing_cluster/README.md covers k3s on plain
VMs, the oop-platform role and what the umbrella chart creates
(databases, Keycloak realms and clients, NATS, NodePorts), how the
encryption keys are protected, where SRM's zones come from, how
federate.yml derives partner records and agreements, what the e2e
suite checks, what undeploy keeps, and the known gaps. It is written to
move unchanged to the Open OP documentation site later. The scenarios
README now points to it instead of repeating it.

inventory/r2.example.yml now matches the inventory the test operators
run on:
- chart branch refactor/helm-2.0, since helm develop still has the
  Release 1 charts (helm!15 brings Release 2);
- one SRM zone per operator through oop_platform_extra_values. SRM
  doesn't discover zones from the cluster (ADR-0059): without a
  bootstrap topology it offers none and agreements fail;
- an oop_platform_partner_agreement example, and plain-HTTP partners
  allowed for test networks.

Every Mermaid diagram renders with mermaid-cli 11.4.2.

Co-Authored-By: default avatarClaude Opus 5.5 <noreply@anthropic.com>
parent a6adf838
Loading
Loading
Loading
Loading
Loading
+2 −0
Changes for README.md: 2 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -16,6 +16,7 @@ This repository contains the public Ansible automation for the ETSI OOP stack.

- `docs/getting-started.md`: environment, inventory, secrets, first deployment
- `docs/deployment.md`: working deployment, undeploy, verification, smoke test flow
- `ansible/playbooks/scenarios/r2_existing_cluster/README.md`: Release 2 on your own clusters (k3s, Helm chart, federation, e2e tests)

## Quick start

@@ -40,6 +41,7 @@ deploying. Set `lite2edge_image` as well when using the `lite2edge` profile.

- `playbooks/scenarios/dual_oop/deploy.yml`: two complete Operator Platforms on two hosts
- `playbooks/scenarios/full_oop/deploy.yml`: one complete Operator Platform on one host
- `playbooks/scenarios/r2_existing_cluster/`: Release 2, one platform per operator on its own cluster, federated and tested; see its `README.md`

## Kubeconfigs

+86 −2
Changes for ansible/inventory/r2.example.yml: 86 added lines, 2 removed lines.
Original line number Diff line number Diff line
@@ -2,6 +2,7 @@
# Example inventory for playbooks/scenarios/r2_existing_cluster: one host per operator.
# Copy to inventory/r2.local.yml (gitignored) and adapt. Secrets go in Ansible Vault, e.g.
# inventory/host_vars/<host>/vault.yml created with `ansible-vault create`.
# What each variable does: playbooks/scenarios/r2_existing_cluster/README.md.

all:
  children:
@@ -19,6 +20,7 @@ all:
          oop_platform_mcc: "214"
          oop_platform_mncs: ["07"]
          oop_platform_federation_clients: [operator-b]
          srm_zone_name: Barcelona

        operator_b:
          ansible_connection: local
@@ -30,6 +32,7 @@ all:
          oop_platform_mcc: "214"
          oop_platform_mncs: ["01"]
          oop_platform_federation_clients: [operator-a]
          srm_zone_name: Madrid

      vars:
        # Published CI builds of each develop branch. Bump them together.
@@ -38,8 +41,80 @@ all:
        oop_platform_srm_image_tag: ci-develop-3200f256
        # No Release 2 portal image is published yet: build one, or turn the portal off.
        oop_platform_portal_enabled: false
        # The R2 charts are not on develop yet.
        oop_platform_chart_version: develop
        # The Release 2 charts are on this branch until oop/deploy/helm!15 merges;
        # helm develop still has the Release 1 (MongoDB) charts.
        oop_platform_chart_version: refactor/helm-2.0
        # Plain HTTP between the operators: test networks only.
        oop_platform_allow_insecure_partner_endpoints: true

        # What every partner in this inventory may use here (federate.yml).
        oop_platform_partner_agreement:
          zones: ["{{ srm_zone_name }}"]
          terms:
            reservedComputeResources:
              - {cpuArchType: ISA_X86_64, numCPU: "2", memory: 4096}
            computeResourceQuotaLimits:
              - {cpuArchType: ISA_X86_64, numCPU: "4", memory: 8192}
            flavoursSupported:
              - flavourId: small
                cpuArchType: ISA_X86_64
                supportedOSTypes:
                  - {architecture: x86_64, distribution: UBUNTU, version: OS_VERSION_UBUNTU_2204_LTS, license: OS_LICENSE_TYPE_FREE}
                numCPU: 1
                memorySize: 1024
                storageSize: 10

        # SRM has no zone until it is given one: here, one zone per operator, with apps
        # deployed by Helm into namespace oop-apps of the operator's own cluster.
        oop_platform_extra_values:
          srm:
            srmcontroller:
              workloads:
                namespaces: [oop-apps]
              controlPaths:
                - control_path_ref: k8s-local
                  adapter: kubernetes
                  config:
                    namespace: oop-apps
              topology: |
                version: openop.srm.bootstrap/v1
                platform_instance_id: {{ oop_platform_federation_id }}
                control_paths:
                  - ref: k8s-local
                    adapter_family: compute
                    adapter_type: kubernetes
                    adapter_contract_version: v1
                    infrastructure:
                      namespace: oop-apps
                zones:
                  - ref: {{ oop_platform_federation_id }}-zone
                    name: {{ srm_zone_name }}
                    kind: resource
                    state: active
                    domains:
                      - ref: {{ oop_platform_federation_id }}-compute
                        name: {{ srm_zone_name }} compute
                        kind: compute
                        state: active
                        capabilities:
                          - ref: query-available-resources
                            name: Query Available Resources
                            kind: query_available_resources
                            state: active
                            control_path_bindings:
                              - ref: query-available-resources-via-k8s-local
                                control_path_ref: k8s-local
                                is_default: true
                                state: active
                          - ref: deploy-workload
                            name: Deploy Workload
                            kind: workload
                            state: active
                            control_path_bindings:
                              - ref: deploy-workload-via-k8s-local
                                control_path_ref: k8s-local
                                is_default: true
                                state: active

        # In the vault, per host (never in plain text):
        # oop_platform_fm_partner_secret_key: <32 bytes, base64url>
@@ -47,3 +122,12 @@ all:
        # oop_platform_postgres_password: ...
        # oop_platform_db_passwords: {keycloak: ..., portal: ..., oeg: ..., srm: ..., fm: ...}
        # oop_platform_keycloak_admin_password: ...

    # Plain Ubuntu VMs: playbooks/infrastructure/k3s/deploy.yml installs k3s first. A host
    # in both groups is the k3s server and the operator host (use ansible_host, not local).
    # k3s_nodes:
    #   hosts:
    #     operator_a:
    #     operator_a_agent:
    #       ansible_host: 192.0.2.11
    #       k3s_server: operator_a
+8 −51
Changes for ansible/playbooks/scenarios/README.md: 8 added lines, 51 removed lines.
Original line number Diff line number Diff line
@@ -37,64 +37,21 @@ ansible-playbook playbooks/scenarios/full_oop_existing_cluster/deploy.yml --limi

### `r2_existing_cluster`

Release 2 (Postgres, NATS, Keycloak realms, portal) on a cluster you already run. Each
host in `oop_platform_operators` is one operator: the `oop-platform` role installs the
`oop-platform` helm chart from `oop/deploy/helm` against that host's kubeconfig, then
checks Keycloak's issuers, OEG and FM through the public address. See
`inventory/r2.example.yml` for the variables each operator needs.
Release 2 (PostgreSQL, NATS, Keycloak realms, portal) on clusters you already run, or on
plain Ubuntu VMs after `playbooks/infrastructure/k3s/deploy.yml`. Each host in
`oop_platform_operators` is one operator with its own cluster. `deploy.yml` installs the
`oop-platform` Helm chart, `federate.yml` registers the operators with each other and
federates them, and `e2e.yml` runs `tests/r2_e2e`. What each step sets up, the variables
and the secrets are in [`r2_existing_cluster/README.md`](r2_existing_cluster/README.md).

```bash
ansible-playbook -i inventory/r2.local.yml playbooks/infrastructure/k3s/deploy.yml   # plain VMs only
ansible-playbook -i inventory/r2.local.yml playbooks/scenarios/r2_existing_cluster/deploy.yml
ansible-playbook -i inventory/r2.local.yml playbooks/scenarios/r2_existing_cluster/undeploy.yml
```

A plain Ubuntu VM gets its cluster first from `playbooks/infrastructure/k3s/deploy.yml`.
It installs single-node k3s at a pinned version (`k3s_version`), Helm and the Python
Kubernetes client, and gives the SSH user a `~/.kube/config`. Put the hosts in a
`k3s_nodes` group. A host that already runs k3s keeps its cluster, version and flags as
they are. `k3s/undeploy.yml` removes the whole cluster with everything in it, so it asks
for `-e k3s_confirm_uninstall=true`.

```bash
ansible-playbook -i inventory/r2.local.yml playbooks/infrastructure/k3s/deploy.yml
```

The chart generates FM's and OEG's encryption keys, the database passwords and the
Keycloak admin password on first install, and keeps them in Secrets in the namespace.
That's fine until the cluster dies, at which point every partner secret FM stored becomes
unreadable. For anything you can't lose, put your own values in Ansible Vault
(`oop_platform_fm_partner_secret_key`, `oop_platform_oeg_callback_secret_key`,
`oop_platform_db_passwords`, ...). The role refuses to swap a key the cluster already
uses, because the data it encrypted would be lost; it prints the command to read the key
in use so you can move that one into the vault instead.

`undeploy.yml` leaves those Secrets and the database volume in place on purpose, so the
next deploy picks up the same keys and data.

Once the operators are up, federate them and run the end-to-end tests:

```bash
ansible-playbook -i inventory/r2.local.yml playbooks/scenarios/r2_existing_cluster/federate.yml
pip install -r tests/r2_e2e/requirements.txt
ansible-playbook -i inventory/r2.local.yml playbooks/scenarios/r2_existing_cluster/e2e.yml
ansible-playbook -i inventory/r2.local.yml playbooks/scenarios/r2_existing_cluster/undeploy.yml
```

`federate.yml` registers each operator in every other one it shares federation clients
with, then establishes the federation both ways. It reads the client secrets from the
partner's `keycloak-credentials` Secret, so it needs access to all the clusters; a partner
you can't reach that way goes in `oop_platform_partners` with its secret from the vault
(see the role defaults). It talks to Federation Manager's internal API through
`kubectl exec` in the FM pod and passes the secrets on stdin. Running it again changes
nothing.

`e2e.yml` runs `tests/r2_e2e` against the inventory: Keycloak issuers, OEG and EWBI
authentication, the federation lifecycle across two operators (including a Federation
Manager restart), and app registration. The tests create throwaway Keycloak clients in
the `oop` realm and delete them afterwards. Known gaps are marked `xfail(strict=True)`
with the reason, so they show up in the report and fail the run as soon as they're
fixed, as a prompt to remove the mark. Pass pytest arguments with
`-e e2e_pytest_args='-k federation'`.

## Undeploy

Each scenario provides:
+394 −0

File added.

Preview size limit exceeded, changes collapsed.