Commit e4ab34cd authored by Dimitrios Gogos's avatar Dimitrios Gogos
Browse files

Refactor federation requirements and design documentation

- Updated REQ-FED-05 to clarify validation of federation agreements and noted that checks are not built yet on SRM's hand-off path.
- Enhanced descriptions for REQ-FED-06, REQ-FED-07, REQ-FED-08, and REQ-FED-09 to indicate proposed features and their current development status.
- Added notes on the implementation status for REQ-FED-11 and REQ-FED-12, highlighting that certain features are not built yet.
- Revised the Partner OP Profile section to reflect that SLA limits are not built yet.
- Clarified the Shared Resource Catalogue section to specify the types of data FM keeps per partner.
- Updated various sections to indicate proposed features and not built yet statuses for certain operations and endpoints.
- Adjusted the error model to include new error types and clarified the mapping of errors to response statuses.
- Documented the retry ownership responsibilities between SRM and FM in the architecture and interface contract documentation.
- Removed references to plain NATS publish in SRM documentation, emphasizing the use of JetStream for publishing with acknowledgements.

This commit improves clarity and accuracy in the documentation regarding the federation system's requirements, design, and current implementation status.
parent d8186e64
Loading
Loading
Loading
Loading
+2 −6
Changes for decisions/0053-srm-fm-federated-lifecycle-handoff-over-the-databus.md: 2 added lines, 6 removed lines.
Original line number Diff line number Diff line
@@ -123,16 +123,12 @@ The envelope keeps the original `operation_id` and `correlation_id`. SRM does no
alone maps to `InstallApp` and `RemoveApp` (ADR-0044). Terminate has no lookup step: FM resolves the
instance from its own `federation_transactions` records.

Three rules make this durable:
Two rules make this durable:

- **`OOP_TASKS` captures `command.fm.>` as well as `command.srm.>`.** FM consumes through a durable
  `fm-task-worker` filtered on `command.fm.>`, beside SRM's `srm-task-worker` on `command.srm.>`.
  A WorkQueue stream only allows several consumers when their filters don't overlap, and these
  don't.
- **Both sides publish through JetStream and wait for the acknowledgement**, never a plain NATS
  publish. A plain publish that nobody is subscribed to at that instant vanishes without an error,
  which would leave the operation pending forever. The same applies to `command.srm.*`.
  FM does this today; **SRM still publishes with a plain NATS publish (not built yet).**
- **FM acknowledges a command only after writing its `federation_transactions` row.** A crash
  before that means redelivery, not a lost deploy.

@@ -249,7 +245,7 @@ needs no federation awareness. Deploy and terminate share a shape.
Harder: `OOP_TASKS` gains a second consumer and `OOP_EVENTS` a second event family, so the stream
definitions in three documents change together. FM becomes a command consumer as well as a
publisher. A bad zone now fails through an event instead of an immediate HTTP error, one hop later.
SRM has to move its publishing onto JetStream, which its `command.srm.*` side needs regardless.
Publishing through JetStream with acknowledgement on the SRM side is a future change.
Federated terminate carries a weaker completion guarantee than local terminate, and that difference
is now explicit on the wire.

+96 −275

File changed.

Preview size limit exceeded, changes collapsed.

+112 −91

File changed.

Preview size limit exceeded, changes collapsed.

+17 −42
Changes for fm/persistence-model.md: 17 added lines, 42 removed lines.
Original line number Diff line number Diff line
@@ -60,7 +60,7 @@ references; it never stores SRM inventory, placement, or capacity.

**PartnerOP** — an external operator platform FM federates with over the EWBI. Holds EWBI base
URL, the partner's OAuth2 client id (the `azp` claim we match inbound tokens against), the client
credentials and token endpoint we use for outbound calls, supported API families, SLA limits, and
credentials and token endpoint we use for outbound calls, supported API families and SLA limits (not built yet), and
connection status. Exactly one current row per partner in `partner_ops`; updates mutate in place
(no immutable version history in Release 2.0).

@@ -80,14 +80,14 @@ URL. Stored in `federation_contexts`.
**RoutingRule** — a mapping from an identifier range (`msisdn_prefix`, `ip_cidr`) to a partner OP,
used for outbound federation routing. Stored in `routing_rules`.

**SharedResourceCatalogue** — edge zones/capabilities a partner has published to us or we to a
partner; lets FM validate that a federated deployment targets an authorized zone. Stored in
`shared_resource_catalogues`.
**SharedResourceCatalogue** — what FM keeps per partner about zones and applications: partner zones,
zone subscriptions in both directions, and application onboardings (ADR-0068). The zones we offer are
computed on request, not stored. Stored in `shared_resource_catalogues`.

**FederationTransaction** — the durable audit record of one EWBI interaction (inbound or outbound).
Carries the directed context, optional federation-side and SRM operation handles, idempotency data,
status, and sanitized request/response summaries. For asynchronous commands it is written `pending`
**before** DataBus publish; synchronous queries are recorded after the HTTP round-trip. Stored in
**before** DataBus publish; a synchronous query is written before the SRM call and completed after it. Stored in
`federation_transactions`.

### Relationships
@@ -113,20 +113,20 @@ anything owned by SRM is held only as a UUID reference.
| 2 | Federation contexts | `federation_contexts` | ✅ Yes | Directed OPG.04 context identity, direction, lifecycle, agreement, and callback URL. |
| 3 | Federation agreements | `federation_agreements` | ✅ Yes | Permissions + `appId → service_specification_id` mappings (ADR-0018); the self-contained federation contract. |
| 4 | Routing rules | `routing_rules` | ✅ Yes | Resolve MSISDN prefix / IP CIDR → partner for outbound routing. |
| 5 | Shared resource catalogues | `shared_resource_catalogues` | ✅ Yes | Zone/capability availability exchanged with partners; validates federated deployment targets. |
| 5 | Shared resource catalogues | `shared_resource_catalogues` | ✅ Yes | Partner zones, zone subscriptions in both directions, and application onboardings (ADR-0068). |
| 6 | Federation transactions | `federation_transactions` | ✅ Yes | Audit + dispute-resolution record; correlates context and internal `operation_id`. |
| 7 | Edge cloud zones | *(none)* | ❌ No | SRM authoritative. FM reads via `GET /internal/zones`; stores only zone UUIDs when catalogued. |
| 8 | Service specifications / manifests | *(none)* | ❌ No | SRM owns the catalog. FM stores only the `service_specification_id` reference inside an agreement. |
| 9 | Service instance detail | *(none)* | ❌ No | SRM owns the instance record. FM keeps only `operation_id`/result summary in a transaction. |
| 10 | Capacity / reservations | *(none)* | ❌ No | SRM holds no stored capacity (ADR-0002); FM's `POST /internal/capacity-check` is best-effort, not persisted. |
| 10 | Capacity / reservations | *(none)* | ❌ No | SRM holds no stored capacity (ADR-0002); FM's best-effort `POST /internal/capacity-check` (not built yet) is not persisted. |

### What is intentionally not persisted

| Concern | Why not persisted |
|---|---|
| Per-partner rate-limit counters | In-process per-pod counters (REQ-FED-11); no shared store in Release 2.0. |
| Per-partner rate-limit counters | Rate limiting is not built yet; when built, counters are in-process per pod (REQ-FED-11). |
| Partner / agreement / routing caches | **Not cached at all** — this data must be strongly consistent (suspending a partner takes effect next request), so it is read from PostgreSQL per request (G1). |
| Raw secret material | `partner_ops` stores secret/path references, not inline client secrets or key bytes. |
| Plaintext secrets | A secret typed in through the registry is stored only encrypted (`partner_op_secrets`, ADR-0067); a mounted secret is referenced by path (`our_client_secret_ref`). |
| SRM inventory, placement, capacity, topology | SRM-owned; FM reads live via `/internal/*` or consumes `event.srm.*`. |
| Intermediate operation progress | Phase 1 tracks `pending → in_progress → terminal`; SRM emits no `in_progress` (K.2). |

@@ -239,7 +239,7 @@ Constraint: `UNIQUE (partner_op_id, direction, federation_context_id)`.
| `id` | UUID | PK, DEFAULT gen_random_uuid() | |
| `partner_op_id` | UUID | NOT NULL, FK → partner_ops(id) | |
| `direction` | VARCHAR(10) | NOT NULL | `inbound` (partner published to us) or `outbound` (we published to partner) |
| `resource_type` | VARCHAR(50) | NOT NULL | `edge_zone`, `network_capability`, etc. |
| `resource_type` | VARCHAR(50) | NOT NULL | `edge_zone`, `zone_subscription`, `accepted_zone`, `app_onboarding`, `file`, `artefact`, `application` (ADR-0068) |
| `resource_data` | JSONB | NOT NULL | OPG.04 zone details/capability metadata, including opaque external `zoneId` and explicit local `zone.id` mapping where we own the zone |
| `synced_at` | TIMESTAMPTZ | NOT NULL, DEFAULT now() | Last sync timestamp |
| `created_at` | TIMESTAMPTZ | NOT NULL, DEFAULT now() | |
@@ -308,7 +308,7 @@ Subjects, streams and consumers are specified in [fm/requirements-and-design.md]
|---|---|---|
| `OOP_TASKS` | `command.srm.service.{deploy,terminate}`, `command.srm.network.capability.{activate,deactivate}` | FM publishes; SRM consumes. Every command carries `CommandEnvelopeV1` with `source=federation`, `federation_partner_ref`, `app_provider_id=partner.id` (ADR-0001, ADR-0018); network-capability payloads carry `source_spec` `family=gsma_opg` (RD §K.2). FM does not publish `service.scale` (no OPG.04 v1.4.0 operation maps to it) or `network.capability.update` (needs an explicit OPG.04 profile/extension first; REQ-FED-16) |
| `OOP_TASKS` | `command.fm.service.{deploy,terminate}` | SRM publishes its federated hand-off; FM consumes via durable `fm-task-worker` (ADR-0053) |
| `OOP_EVENTS` | `event.srm.operation.status`, `event.srm.operation.completed`, `event.srm.catalog.updated` | FM consumes via durable `fm-event-worker` (explicit ack, max-deliver 3) |
| `OOP_EVENTS` | `event.srm.operation.status`, `event.srm.operation.completed`, `event.srm.catalog.updated` | FM consumes via durable `fm-event-worker` (explicit ack, max-deliver 3); only `event.srm.operation.completed` is acted on — `event.srm.operation.status` and the catalogue re-sync on `event.srm.catalog.updated` are not built yet |
| `OOP_EVENTS` | `event.fm.operation.completed` | FM publishes the hand-off outcome; SRM consumes it (ADR-0053) |

Operator-triggered federation lifecycle reaches FM as synchronous HTTP from OEG (ADR-0051); no
@@ -320,36 +320,10 @@ on successful publish it sets `in_progress` locally. Duplicates are absorbed by

## O. Implementation Translation

FM follows the same hexagonal layout as OEG/SRM (ADR-0016; single Python/FastAPI service).
FM follows the same hexagonal layout as OEG/SRM (ADR-0016; single Python/FastAPI service). The module layout is in
[fm/implementation-and-delivery.md](/fm/implementation-and-delivery.md) §O.

```
federation-manager/
├── src/federation_manager/
│   ├── core/                     # config, logging
│   ├── domain/
│   │   ├── models.py             # PartnerOP, Context, Agreement, RoutingRule, Transaction entities + rules
│   │   ├── ports.py              # Protocols for repos, SRM client, DataBus publisher
│   │   ├── errors.py             # federation errors (no HTTP detail)
│   │   ├── topology_hiding.py    # sanitize rules
│   │   └── routing.py            # MSISDN prefix / IP CIDR resolution
│   ├── application/              # inbound / outbound / partner / agreement / catalogue services
│   ├── adapters/
│   │   ├── database/             # SQLAlchemy async: engine, tables, per-entity repos
│   │   ├── databus/              # command.srm.* publisher, event.srm.* / command.fm.* consumers
│   │   ├── http/                 # SRM internal client, outbound EWBI client
│   │   └── security/             # Keycloak JWT/JWKS validation, outbound token provider
│   ├── api/
│   │   ├── ewbi/v1/              # partner-facing OPG.04 routes
│   │   ├── internal/             # operator-facing admin routes, OEG/SRM hand-offs
│   │   ├── security.py           # OAuth2 scheme + Bearer extraction
│   │   └── platform/health.py
│   ├── middleware/               # topology_hiding
│   ├── dependencies.py           # DI wiring (repo/adapter selection)
│   └── main.py                   # composition root + lifespan
└── tests/  unit | integration | e2e
```

- **SQLAlchemy 2.0 async** session for `fm_db`; schema-init-on-startup for Release 2 (Alembic deferred).
- **SQLAlchemy 2.0 async** session for `fm_db`; the schema is managed by Alembic migrations, applied at startup.
- **NATS JetStream** publisher/durable consumer via `nats-py`.
- **Unit of work:** persist `federation_transactions` (`pending`) **before** publish; on publish set
  `in_progress`; finalize from `event.srm.operation.completed`.
@@ -363,10 +337,10 @@ federation-manager/
| Topic | Approach |
|---|---|
| Inbound AuthN/Z | OAuth2 Client-Credentials + signed JWT via Keycloak `federation` realm (`fed-mgmt` scope); FM validates against Keycloak's JWKS (ADR-0042) |
| Admin AuthN/Z | Keycloak JWT, `operator` role, on `/internal/*` admin routes only — same centralized Keycloak, `oop` realm |
| Admin AuthN/Z | No token check in FM on `/internal/*`: the routes are reachable only inside the cluster, and OEG authenticates the operator (`operator` role) on the routes it relays (ADR-0051) |
| Outbound AuthN | OAuth2 Client-Credentials — FM fetches a token from the partner's realm using our registered client credentials (ADR-0042) |
| Future: mTLS defense-in-depth | Deferred until real Nginx mTLS termination exists (ADR-0042). `partner_ca_cert_path` supports ordinary HTTPS verification; it is not an mTLS identity field. |
| Topology hiding | mandatory `TopologyHider` on every EWBI response (REQ-FED-02); strips internal IPs, cluster/namespace/node names, pod UIDs, trace IDs |
| Topology hiding | mandatory `TopologyHidingMiddleware` on every EWBI response (REQ-FED-02); strips internal IPs, cluster/namespace/node names, pod UIDs, trace IDs |
| Correlation boundary | internal `correlation_id` stays in logs and `federation_transactions`; no custom header or field crosses the EWBI; partner-side correlation uses OPG.04 identifiers (`federationContextId`, `txnIdentifier`, `Idempotency-Key`) |
| Header trust | FM must be reachable only via Nginx. FM trusts no caller-supplied identity header — partner identity comes solely from the validated JWT. |

@@ -396,3 +370,4 @@ What FM `develop` has today; update as the schema changes.
| 2.1-draft | 2026-09-08 | `federation_contexts` landed in code; status vocabulary enumerated from the OPG.04 `Status` enum. |
| 2.1-draft | 2026-09-08 | `external_resource_id` records the wire spelling of the external identifier; LCM audit columns (idempotency key, request fingerprint, callback) landed in code. |
| 2.1-draft | 2026-10-13 | Single source for FM's schema: §J.1 takes over the table definitions from requirements-and-design §K.1, adding `partner_op_secrets` (ADR-0067) and `federation_agreements.zone_resources`. §J.2 subjects corrected for ADR-0051 and ADR-0053. Implementation status brought up to date with FM `develop`. |
| 2.1-draft | 2026-10-13 | Full audit against FM `develop`: shared-resource types, synchronous-query audit order, secret storage, Alembic, operator access to `/internal/*` and topology-hiding names corrected; module tree moved to implementation-and-delivery §O; partner SLA/API columns and rate limiting marked not built yet. |
+79 −54

File changed.

Preview size limit exceeded, changes collapsed.

Loading