Merge branch 'fm-srm-handoff' into 'develop'

# ADR-0018: Federation tenancy — the partner OP is the tenant; federation agreements map partner requests to local service specifications

**Status:** accepted

**Status:** accepted — the manual app mapping is now the fallback: an application the partner onboarded over the EWBI is resolved from that onboarding first ([ADR-0068](/decisions/0068-fm-onboards-apps-at-partners-over-ewbi-before-installapp.md)). The partner-as-tenant decision is unchanged.

**Date:** 2026-07-05

**Affects:** [fm/overview-and-architecture.md](/fm/overview-and-architecture.md), [fm/requirements-and-design.md](/fm/requirements-and-design.md), [fm/implementation-and-delivery.md](/fm/implementation-and-delivery.md)

---

type: Decision Record

title: "ADR-0053: SRM hands federated deploys and terminates to FM over the DataBus, and FM reports the outcome the same way"

description: "SRM hands federated deploys and terminates to FM over the DataBus, and FM reports the outcome the same way"

tags: [adr]

timestamp: 2026-09-22T00:00:00Z

status: "proposed"

---

# ADR-0053: SRM hands federated deploys and terminates to FM over the DataBus, and FM reports the outcome the same way

**Status:** proposed (amended 2026-09-25: SRM keeps instance rows for partner targets, superseded on 2026-10-08; 2026-09-29: `partner_not_found`; 2026-10-08: order-local `federated_handoff` instead of partner inventory)

**Date:** 2026-09-22

**Affects:** [fm/overview-and-architecture.md](/fm/overview-and-architecture.md) (open item G4, UC-FM-07), [fm/requirements-and-design.md](/fm/requirements-and-design.md) (the SRM→FM hand-off section §L.3a, FM's DataBus sections §K.2 and §M), [srm/interface-contract.md](/srm/interface-contract.md) (stream configuration §D, the federated hand-off flow §F.4a), [srm/persistence-model.md](/srm/persistence-model.md) (`service_order.federated_handoff`), [platform/architecture.md](/platform/architecture.md) (DataBus configuration)

**Amends:** ADR-0001 and ADR-0022, where they make SRM the only consumer of `OOP_TASKS`; ADR-0046, where it keeps the DataBus to `command.srm.*` / `event.srm.*`

> **Numbering.** `0052` is unused.

## Context

ADR-0044 made SRM the component that decides whether a deploy's target zone is ours or a partner's,

and made it hand partner-bound work to FM. How that hand-off travels was left open, as item G4 in

the FM overview document, which leaned towards synchronous internal HTTP. The first version of this

record followed that lean for the request and added an event for the result. Review with the OEG

and SRM owners moved the request onto the DataBus as well, for the reasons below.

Five facts shape the decision.

**OPG.04 gives no result on the request.** In the GSMA OPG.04 v6.0 machine contract

(`OPG.04_EWBI_Federation_API_v1.4.0.yaml`), `InstallApp`'s only success response is `202`, and the

outcome arrives later as an `onInstanceStatusEvent` callback to the `appInstCallbackLink` the

request carries. `RemoveApp` answers `200` and defines no callback at all; `GetAppInstanceDetails`

reports `PENDING | READY | FAILED | TERMINATING`, with no value for a finished removal. Whatever

carries the request, the outcome needs its own channel, and for termination there's no outcome to

observe beyond the partner saying yes.

**FM can't call SRM back over HTTP.** SRM's interface contract says SRM "exposes no public HTTP

lifecycle mutation endpoints", and completing an operation is exactly that. So the result has to be

an event.

**Nothing upstream waits on the hand-off.** OEG answers the App Provider's `POST /appinstances` with

`202` as soon as it has stored the operation and published `command.srm.service.deploy`; `DELETE`

works the same way. By the time SRM decides a zone belongs to a partner, the App Provider already

has their `appInstanceId`. An HTTP `202` from FM would only tell SRM that FM took the job, and a

JetStream publish acknowledgement tells it the same thing.

**One hand-off, one transport.** With the result on the bus anyway, an HTTP request leaves SRM

and FM speaking two transports for a single piece of work, with two sets of retry, timeout and

idempotency rules to keep straight. SRM's own work-in-progress FM client (`refactor/fm-client`)

already publishes the request as a command, which says something about what the SRM side finds

natural.

**Nobody records a federated operation today.** The platform's Component Boundary Matrix gives

"Operation lifecycle state management" to SRM (service orders, instances) and OEG (operations), and

names FM as not an owner; FM's persistence model agrees, keeping only a UUID reference to anything

SRM owns. But the SRM hand-off flow says "No local `service_order`/`service_instance` rows are

written for the federated portion", which leaves a federated deploy recorded by no one: no

idempotency anchor, and no way for SRM to match a late result after a restart.

Options considered:

- **A. HTTP both ways**, FM calling SRM to complete the operation. Rejected: breaks SRM's rule

  against HTTP lifecycle mutations.

- **B. HTTP request, DataBus result** (this record's first version). Rejected in review: two

  transports for one hand-off, and the synchronous `202` carries nothing SRM acts on. Its one real

  advantage, refusing a bad zone instantly, saves a round trip on an error path that's rare once

  SRM has already looked the zone up.

- **C. DataBus both ways.** Chosen.

- **D. SRM polls FM for the outcome.** Rejected: turns a push into a poll across a component

  boundary, and OPG.04 bounds neither callback latency nor removal time.

## Decision

SRM and FM exchange lifecycle work only over the DataBus, a command in each direction's natural

stream. Reads stay HTTP.

```mermaid

sequenceDiagram

    autonumber

    actor AP as App Provider

    box Our operator

        participant OEG

        participant SRMA as SRM-A

        participant FMA as FM-A

    end

    box Partner operator

        participant FMB as FM-B

    end

    AP->>OEG: POST /appinstances (CAMARA)

    OEG-->>AP: 202 appInstanceId

    OEG->>SRMA: command.srm.service.deploy [OOP_TASKS]

    SRMA->>FMA: GET /internal/federated-zones/{zone_id}

    FMA-->>SRMA: 200 zone is federated

    Note over SRMA: writes service_order row

    SRMA->>FMA: command.fm.service.deploy [OOP_TASKS]

    Note over FMA: writes federation_transactions row

    FMA->>FMB: InstallApp (GSMA EWBI)

    FMB-->>FMA: 202 appInstIdentifier

    FMB->>FMA: onInstanceStatusEvent (GSMA callback)

    FMA->>SRMA: event.fm.operation.completed, partner_confirmed [OOP_EVENTS]

    SRMA->>OEG: event.srm.operation.completed [OOP_EVENTS]

    OEG->>AP: callback, or App Provider polls

```

### Reads stay synchronous HTTP

| Endpoint | Used for |

|---|---|

| `GET /internal/federated-zones/{zone_id}` | Deploy: SRM checks whether a zone missing from its inventory belongs to a partner. `404` means nobody has it, an ordinary placement failure |

| `GET /internal/federated-zones` | SRM's zone listing, so partner zones show up alongside local ones |

A read has an immediate answer and changes nothing, so request/reply over HTTP fits it. JetStream

isn't built for request/reply.

### The request: `command.fm.service.deploy` and `command.fm.service.terminate`

| Subject | Stream | Payload | SRM publishes it when |

|---|---|---|---|

| `command.fm.service.deploy` | `OOP_TASKS` | the `SrmServiceDeployV1` SRM received, unshaped | the target zone came back federated |

| `command.fm.service.terminate` | `OOP_TASKS` | the `SrmServiceTerminateV1` SRM received, unshaped | the instance is absent from SRM's local inventory (partner-hosted; SRM never wrote a partner `service_instance`) |

The envelope keeps the original `operation_id` and `correlation_id`. SRM does no GSMA shaping; FM

alone maps to `InstallApp` and `RemoveApp` (ADR-0044). Terminate has no lookup step: FM resolves the

instance from its own `federation_transactions` records.

Two rules make this durable:

- **`OOP_TASKS` captures `command.fm.>` as well as `command.srm.>`.** FM consumes through a durable

  `fm-task-worker` filtered on `command.fm.>`, beside SRM's `srm-task-worker` on `command.srm.>`.

  A WorkQueue stream only allows several consumers when their filters don't overlap, and these

  don't.

- **FM acknowledges a command only after writing its `federation_transactions` row.** A crash

  before that means redelivery, not a lost deploy.

FM keys on `operation_id`: a redelivered command whose `operation_id` already has a transaction row

starts no second EWBI exchange.

**Refusals come back as a failed result.** When FM can't act (no partner offers the zone, the

agreement forbids it, the partner is suspended, the instance is unknown), it publishes

`event.fm.operation.completed` with `status: failed` and the problem type in `error`. When the

partner's EWBI endpoint is unreachable, FM negatively acknowledges the command so JetStream

redelivers it, and publishes the failure only once deliveries run out.

**Not built yet:** FM does not check the agreement or the partner's status on this path, and does

not publish the failure after the last delivery; an unreachable partner leaves the command

unacknowledged until JetStream stops redelivering it, and the operation stays pending.

### The result: `event.fm.operation.completed` on `OOP_EVENTS`

`OOP_EVENTS` captures `event.fm.>` as well as `event.srm.>`. FM publishes one terminal event per

`operation_id`; SRM consumes it, completes its `service_order`, and publishes

`event.srm.operation.completed` exactly as it would for a local deploy. OEG, and through it the App

Provider, can't tell the two apart, which topology hiding requires anyway.

For OEG to update its record, SRM's event has to list the instance in `instances[]`. OEG's

completion handler only changes an instance it finds there, so a successful event with an empty

list would leave the App Provider's instance `INSTANTIATING` or `TERMINATING` for good.

The event fires at different moments for the two operations, because OPG.04 guarantees different

things:

| `metadata.completion_basis` | Operation | FM publishes when | Means |

|---|---|---|---|

| `partner_confirmed` | deploy | the partner's `onInstanceStatusEvent` arrives | the instance is running |

| `deadline_expired` | deploy | no callback before the deadline | failed; the partner never answered (not built yet, see below) |

| `partner_accepted` | terminate | the partner answers `RemoveApp` with `200` | the partner said yes; removal isn't confirmed |

| `partner_not_found` | terminate | the partner answers `RemoveApp` with `404` | the partner no longer has the instance; the terminate is complete |

Termination gets the weaker claim because OPG.04 offers nothing stronger to observe. We don't build

machinery to chase a partner who accepts a removal and then leaves the app running: that partner

keeps paying for compute on a workload it can no longer bill for once the agreement's usage record

closes, so cleaning up is in its own interest. Reconciliation stays as the backstop, through SRM's

existing `event.srm.inventory.drift_detected` family once federated instances become visible to it.

### The deploy callback deadline

**Not built yet:** FM sets no deadline today, so a deploy whose partner never calls back stays pending.

FM waits **30 minutes** by default for `onInstanceStatusEvent`. It's configuration, and a federation

agreement may later carry a per-partner value. It's long on purpose: a cold edge node pulling a

multi-gigabyte AI or GPU image can take many minutes before the instance runs, and a deadline that

fires during an ordinary slow deploy would tell the App Provider their app failed while it's running

on the partner's hardware.

FM computes the deadline from the stored `federation_transactions.started_at`, never from an

in-memory timer, so a restart doesn't lose track of it.

### Record ownership

This amends step 4 of the SRM hand-off flow ("No local `service_order`/`service_instance` rows are

written for the federated portion"), and **supersedes the 2026-09-25 amendment** that had SRM

persist partner `service_instance` rows and `federation_business` zone rows:

- **SRM writes a `service_order` row** for federated deploys and terminates, as for any command it

  accepts. Its `UNIQUE(operation_id)` constraint is the idempotency anchor, and the row lets SRM

  match a late `event.fm.operation.completed` after a restart. The Component Boundary Matrix already

  gives service orders to SRM.

- **`payload_snapshot` stays a frozen copy of the inbound command** (audit only). It is not mutated

  to track partner progress.

- **Pending partner targets live only in `service_order.federated_handoff`** (JSONB, default `{}`).

  Shape when the order has federated placements:

  ```json

  {

    "targets": [

      {

        "app_instance_id": "<uuid>",

        "zone_id": "<uuid>",

        "status": "pending|completed|failed"

      }

    ]

  }

  ```

  Empty `{}` when the order has no federated targets. SRM does **not** persist partner zones in

  `zone` or partner `service_instance` rows; partner inventory stays with FM

  (`federation_transactions` / `shared_resource_catalogues`).

- **SRM completes the order from FM's answer.** The order stays `in_progress` while any

  `federated_handoff.targets[]` entry is `pending`. Each `event.fm.operation.completed` updates the

  matching target's `status`; once none is left `pending`, SRM completes the order (`completed`,

  `partially_completed` or `failed`) and publishes `event.srm.operation.completed` whose

  `instances[]` lists local inventory rows plus the partner entries from `federated_handoff`

  (so OEG can update every `appInstanceId`).

FM keeps its `federation_transactions` row unchanged, as the audit and dispute record that ties the

partner's `appInstIdentifier` to our `operation_id`.

### What this amends

- **ADR-0001 and ADR-0022** make SRM the only consumer of `OOP_TASKS`. SRM stays the only consumer

  of `command.srm.>`; FM becomes the consumer of `command.fm.>`.

- **ADR-0046** says the `command.fm.federation.*` family is not created and the DataBus "stays

  scoped to the `command.srm.*` / `event.srm.*` flows FM already uses". The first half stands:

  operator-triggered federation lifecycle remains synchronous HTTP from OEG, and no

  `command.fm.federation.*` subject exists. The second half now also covers the SRM↔FM lifecycle

  hand-off, `command.fm.service.*` and `event.fm.*`.

## Consequences

Easier: SRM and FM share one transport and one set of delivery rules for lifecycle work, and FM's

consumer gets JetStream's redelivery and dead-lettering for free. Nothing holds a connection open

across a partner exchange. Local and federated fulfilment end in the same terminal event, so OEG

needs no federation awareness. Deploy and terminate share a shape.

Harder: `OOP_TASKS` gains a second consumer and `OOP_EVENTS` a second event family, so the stream

definitions in three documents change together. FM becomes a command consumer as well as a

publisher. A bad zone now fails through an event instead of an immediate HTTP error, one hop later.

Publishing through JetStream with acknowledgement on the SRM side is a future change.

Federated terminate carries a weaker completion guarantee than local terminate, and that difference

is now explicit on the wire.

Revisit when the runtime coordinator described in the platform's future-architecture document

(§6.1) gets built: it's meant to own routing between local and federated fulfilment and would take

over both hops.

---

type: Decision Record

title: "ADR-0054: FM receives every GSMA callback through one router under /operatorplatform/federation/v1/callbacks/{kind}"

description: "FM receives every GSMA callback through one router under /operatorplatform/federation/v1/callbacks/{kind}"

tags: [adr]

timestamp: 2026-09-22T00:00:00Z

status: "proposed"

---

# ADR-0054: FM receives every GSMA callback through one router under `/operatorplatform/federation/v1/callbacks/{kind}`

**Status:** proposed

**Date:** 2026-09-22

**Affects:** [fm/requirements-and-design.md](/fm/requirements-and-design.md) (new §L.1a, callbacks FM receives), [fm/overview-and-architecture.md](/fm/overview-and-architecture.md) (partner authentication)

## Context

When FM federates with a partner as the originating OP, several OPG.04 operations make the partner

call FM back later: `CreateFederation` hands over a `partnerStatusLink`, `InstallApp` an

`appInstCallbackLink`, `ZoneSubscribe` an `availZoneNotifLink`. OPG.04 v6.0 (artifact v1.4.0)

defines eleven such callbacks, and every one has the same shape: a `POST` to a URL the originating

OP chose, authorised with the `notifClientCredentials` scheme and the `fed-mgmt-notif` scope.

FM already sends callbacks as the partner (`onInstanceStatusEvent` after an inbound `InstallApp`),

but it receives none. It hands every partner a `partnerStatusLink` today (`FM_PARTNER_STATUS_LINK`,

default `/operatorplatform/federation/v1/partner-status`) and nothing listens there, so a partner's

`onPartnerStatusEvent` gets a `404`. The deploy hand-off in ADR-0053 depends on FM receiving

`onInstanceStatusEvent` too.

Two more gaps turned up while checking this. REQ-FED-01 says FM validates `fed-mgmt-notif` on

inbound calls, but `PartnerAuthenticator` hardcodes `fed-mgmt`. And no `federation` realm defines a

`fed-mgmt-notif` scope, neither FM's dev realm nor the platform Keycloak chart in helm MR

oop/deploy/helm!12, so a partner can't obtain a callback token from us, and FM's own callback client

can't obtain one from a partner running the same setup.

## Decision

FM receives all GSMA callbacks through one router, and every callback goes through the same three

steps before a per-kind handler runs.

```mermaid

flowchart LR

    P[Partner OP] -->|"POST /operatorplatform/federation/v1/callbacks/{kind}<br/>Bearer token, scope fed-mgmt-notif"| R

    subgraph FM ["FM, originating side"]

        R[Callback router] --> A["Authenticate<br/>scope fed-mgmt-notif<br/>azp to an active partner"]

        A --> C["Resolve context<br/>federationContextId must be an<br/>outbound context of that partner"]

        C --> D{kind}

        D -->|partner-status| H1["onPartnerStatusEvent<br/>federation_contexts.status"]

        D -->|instance-status| H2["onInstanceStatusEvent<br/>federation_transactions"]

        D -->|zone-resource| H3["onZoneResourceUpdateEvent<br/>shared_resource_catalogues"]

    end

```

**URLs.** Callbacks live under `/operatorplatform/federation/v1/callbacks/{kind}`, with one `kind`

per callback: `partner-status`, `instance-status`, `zone-resource`, and more as FM starts calling the

operations that trigger them. OPG.04 leaves the URL to the originating OP; putting it under the

partner-facing prefix means the ingress already routes it to FM, with no new rule. The

`FM_PARTNER_STATUS_LINK` default moves to `/operatorplatform/federation/v1/callbacks/partner-status`.

Nothing received on the old URL, so the move breaks nothing.

**Authentication.** A callback needs a token from our `federation` realm carrying `fed-mgmt-notif`

(REQ-FED-01). A `fed-mgmt` token alone gets `401`: the two scopes separate "you may call our API"

from "you may notify us", which is the distinction OPG.04 draws. The `azp` claim resolves to an

active partner exactly as for other EWBI calls.

**Context check.** Every callback payload carries `federationContextId`. It must name an

**outbound** federation context belonging to the calling partner, meaning one FM created with them.

Anything else gets `404`, so no partner can touch a federation it isn't party to.

**Handlers** are idempotent, because partners may retry, and answer with the status code OPG.04 gives

each callback (`204` for `onPartnerStatusEvent` and `onInstanceStatusEvent`, `200` for

`onZoneResourceUpdateEvent`).

**Order of work.** `onPartnerStatusEvent` came first: it updates `federation_contexts.status`, whose

values mirror the callback's `Status` enum. `onInstanceStatusEvent` followed with FM's outbound

`InstallApp`, which creates the transaction it updates, and publishes `event.fm.operation.completed`

(ADR-0053). Both are built. `onZoneResourceUpdateEvent` is not built yet.

**Keycloak.** The `federation` realm defines a `fed-mgmt-notif` client scope and attaches it to each

partner client as an **optional** scope, beside the default `fed-mgmt`. Optional means a token only

carries it when the client asks, so an ordinary API token can't double as a callback token. This

applies to FM's dev realm and to the platform Keycloak chart.

## Consequences

Easier: each new callback is one handler, not a new endpoint with its own auth and ownership checks.

A partner can finally report federation status changes to us. The receiving side of ADR-0053's

deploy flow has a place to live.

Harder: the platform Keycloak chart needs a second client scope per partner client, which its

current template (one `clientScope` value) doesn't support. Partners federating with us have to

request `fed-mgmt-notif` when they call us back, which OPG.04 already requires of them.