Merge branch 'fm-srm-handoff' into 'docs/fm-internal-federation-statuses'

---

type: Decision Record

title: "ADR-0053: SRM hands federated deploys and terminates to FM over the DataBus, and FM reports the outcome the same way"

description: "SRM hands federated deploys and terminates to FM over the DataBus, and FM reports the outcome the same way"

tags: [adr]

timestamp: 2026-09-22T00:00:00Z

status: "proposed"

---

# ADR-0053: SRM hands federated deploys and terminates to FM over the DataBus, and FM reports the outcome the same way

**Status:** proposed (amended 2026-09-25: SRM keeps instance rows for partner targets)

**Date:** 2026-09-22

**Affects:** [fm/overview-and-architecture.md](/fm/overview-and-architecture.md) (open item G4, UC-FM-07), [fm/requirements-and-design.md](/fm/requirements-and-design.md) (the SRM→FM hand-off section §L.3a, FM's DataBus sections §K.2 and §M), [srm/interface-contract.md](/srm/interface-contract.md) (stream configuration §D, the federated hand-off flow §F.4a), [platform/architecture.md](/platform/architecture.md) (DataBus configuration)

**Amends:** ADR-0001 and ADR-0022, where they make SRM the only consumer of `OOP_TASKS`; ADR-0046, where it keeps the DataBus to `command.srm.*` / `event.srm.*`

> **Numbering.** `0052` is skipped on purpose. `oeg-fm` has renumbered its ADRs once already

> (`0047`/`0048` became `0050`/`0051`), and the gap means a second renumber there can't land on

> this record.

## Context

ADR-0044 made SRM the component that decides whether a deploy's target zone is ours or a partner's,

and made it hand partner-bound work to FM. How that hand-off travels was left open, as item G4 in

the FM overview document, which leaned towards synchronous internal HTTP. The first version of this

record followed that lean for the request and added an event for the result. Review with the OEG

and SRM owners moved the request onto the DataBus as well, for the reasons below.

Five facts shape the decision.

**OPG.04 gives no result on the request.** In the GSMA OPG.04 v6.0 machine contract

(`OPG.04_EWBI_Federation_API_v1.4.0.yaml`), `InstallApp`'s only success response is `202`, and the

outcome arrives later as an `onInstanceStatusEvent` callback to the `appInstCallbackLink` the

request carries. `RemoveApp` answers `200` and defines no callback at all; `GetAppInstanceDetails`

reports `PENDING | READY | FAILED | TERMINATING`, with no value for a finished removal. Whatever

carries the request, the outcome needs its own channel, and for termination there's no outcome to

observe beyond the partner saying yes.

**FM can't call SRM back over HTTP.** SRM's interface contract says SRM "exposes no public HTTP

lifecycle mutation endpoints", and completing an operation is exactly that. So the result has to be

an event.

**Nothing upstream waits on the hand-off.** OEG answers the App Provider's `POST /appinstances` with

`202` as soon as it has stored the operation and published `command.srm.service.deploy`; `DELETE`

works the same way. By the time SRM decides a zone belongs to a partner, the App Provider already

has their `appInstanceId`. An HTTP `202` from FM would only tell SRM that FM took the job, and a

JetStream publish acknowledgement tells it the same thing.

**One hand-off, one transport.** With the result on the bus anyway, an HTTP request leaves SRM

and FM speaking two transports for a single piece of work, with two sets of retry, timeout and

idempotency rules to keep straight. SRM's own work-in-progress FM client (`refactor/fm-client`)

already publishes the request as a command, which says something about what the SRM side finds

natural.

**Nobody records a federated operation today.** The platform's Component Boundary Matrix gives

"Operation lifecycle state management" to SRM (service orders, instances) and OEG (operations), and

names FM as not an owner; FM's persistence model agrees, keeping only a UUID reference to anything

SRM owns. But the SRM hand-off flow says "No local `service_order`/`service_instance` rows are

written for the federated portion", which leaves a federated deploy recorded by no one: no

idempotency anchor, and no way for SRM to match a late result after a restart.

Options considered:

- **A. HTTP both ways**, FM calling SRM to complete the operation. Rejected: breaks SRM's rule

  against HTTP lifecycle mutations.

- **B. HTTP request, DataBus result** (this record's first version). Rejected in review: two

  transports for one hand-off, and the synchronous `202` carries nothing SRM acts on. Its one real

  advantage, refusing a bad zone instantly, saves a round trip on an error path that's rare once

  SRM has already looked the zone up.

- **C. DataBus both ways.** Chosen.

- **D. SRM polls FM for the outcome.** Rejected: turns a push into a poll across a component

  boundary, and OPG.04 bounds neither callback latency nor removal time.

## Decision

SRM and FM exchange lifecycle work only over the DataBus, a command in each direction's natural

stream. Reads stay HTTP.

```mermaid

sequenceDiagram

    autonumber

    actor AP as App Provider

    box Our operator

        participant OEG

        participant SRMA as SRM-A

        participant FMA as FM-A

    end

    box Partner operator

        participant FMB as FM-B

    end

    AP->>OEG: POST /appinstances (CAMARA)

    OEG-->>AP: 202 appInstanceId

    OEG->>SRMA: command.srm.service.deploy [OOP_TASKS]

    SRMA->>FMA: GET /internal/federated-zones/{zone_id}

    FMA-->>SRMA: 200 zone is federated

    Note over SRMA: writes service_order row

    SRMA->>FMA: command.fm.service.deploy [OOP_TASKS]

    Note over FMA: writes federation_transactions row

    FMA->>FMB: InstallApp (GSMA EWBI)

    FMB-->>FMA: 202 appInstIdentifier

    FMB->>FMA: onInstanceStatusEvent (GSMA callback)

    FMA->>SRMA: event.fm.operation.completed, partner_confirmed [OOP_EVENTS]

    SRMA->>OEG: event.srm.operation.completed [OOP_EVENTS]

    OEG->>AP: callback, or App Provider polls

```

### Reads stay synchronous HTTP

| Endpoint | Used for |

|---|---|

| `GET /internal/federated-zones/{zone_id}` | Deploy: SRM checks whether a zone missing from its inventory belongs to a partner. `404` means nobody has it, an ordinary placement failure |

| `GET /internal/federated-zones` | SRM's zone listing, so partner zones show up alongside local ones |

A read has an immediate answer and changes nothing, so request/reply over HTTP fits it. JetStream

isn't built for request/reply.

### The request: `command.fm.service.deploy` and `command.fm.service.terminate`

| Subject | Stream | Payload | SRM publishes it when |

|---|---|---|---|

| `command.fm.service.deploy` | `OOP_TASKS` | the `SrmServiceDeployV1` SRM received, unshaped | the target zone came back federated |

| `command.fm.service.terminate` | `OOP_TASKS` | the `SrmServiceTerminateV1` SRM received, unshaped | the instance's row says `hosted_by: partner` |

The envelope keeps the original `operation_id` and `correlation_id`. SRM does no GSMA shaping; FM

alone maps to `InstallApp` and `RemoveApp` (ADR-0044). Terminate has no lookup step: FM resolves the

instance from its own `federation_transactions` records.

Three rules make this durable:

- **`OOP_TASKS` captures `command.fm.>` as well as `command.srm.>`.** FM consumes through a durable

  `fm-task-worker` filtered on `command.fm.>`, beside SRM's `srm-task-worker` on `command.srm.>`.

  A WorkQueue stream only allows several consumers when their filters don't overlap, and these

  don't.

- **Both sides publish through JetStream and wait for the acknowledgement**, never a plain NATS

  publish. A plain publish that nobody is subscribed to at that instant vanishes without an error,

  which would leave the operation pending forever. The same applies to `command.srm.*`.

- **FM acknowledges a command only after writing its `federation_transactions` row.** A crash

  before that means redelivery, not a lost deploy.

FM keys on `operation_id`: a redelivered command whose `operation_id` already has a transaction row

starts no second EWBI exchange.

**Refusals come back as a failed result.** When FM can't act (no partner offers the zone, the

agreement forbids it, the partner is suspended, the instance is unknown), it publishes

`event.fm.operation.completed` with `status: failed` and the problem type in `error`. When the

partner's EWBI endpoint is unreachable, FM negatively acknowledges the command so JetStream

redelivers it, and publishes the failure only once deliveries run out.

### The result: `event.fm.operation.completed` on `OOP_EVENTS`

`OOP_EVENTS` captures `event.fm.>` as well as `event.srm.>`. FM publishes one terminal event per

`operation_id`; SRM consumes it, completes its `service_order`, and publishes

`event.srm.operation.completed` exactly as it would for a local deploy. OEG, and through it the App

Provider, can't tell the two apart, which topology hiding requires anyway.

For OEG to update its record, SRM's event has to list the instance in `instances[]`. OEG's

completion handler only changes an instance it finds there, so a successful event with an empty

list would leave the App Provider's instance `INSTANTIATING` or `TERMINATING` for good.

The event fires at different moments for the two operations, because OPG.04 guarantees different

things:

| `metadata.completion_basis` | Operation | FM publishes when | Means |

|---|---|---|---|

| `partner_confirmed` | deploy | the partner's `onInstanceStatusEvent` arrives | the instance is running |

| `deadline_expired` | deploy | no callback before the deadline | failed; the partner never answered |

| `partner_accepted` | terminate | the partner answers `RemoveApp` with `200` | the partner said yes; removal isn't confirmed |

Termination gets the weaker claim because OPG.04 offers nothing stronger to observe. We don't build

machinery to chase a partner who accepts a removal and then leaves the app running: that partner

keeps paying for compute on a workload it can no longer bill for once the agreement's usage record

closes, so cleaning up is in its own interest. Reconciliation stays as the backstop, through SRM's

existing `event.srm.inventory.drift_detected` family once federated instances become visible to it.

### The deploy callback deadline

FM waits **30 minutes** by default for `onInstanceStatusEvent`. It's configuration, and a federation

agreement may later carry a per-partner value. It's long on purpose: a cold edge node pulling a

multi-gigabyte AI or GPU image can take many minutes before the instance runs, and a deadline that

fires during an ordinary slow deploy would tell the App Provider their app failed while it's running

on the partner's hardware.

FM computes the deadline from the stored `federation_transactions.started_at`, never from an

in-memory timer, so a restart doesn't lose track of it.

### Record ownership

This amends step 4 of the SRM hand-off flow ("No local `service_order`/`service_instance` rows are

written for the federated portion"), which splits in two:

- **SRM writes a `service_order` row** for federated deploys and terminates, as for any command it

  accepts. Its `UNIQUE(operation_id)` constraint is the idempotency anchor, and the row lets SRM

  match a late `event.fm.operation.completed` after a restart. The Component Boundary Matrix already

  gives service orders to SRM.

- **SRM writes a `service_instance` row for each partner target too**, in state `creating` with

  `metadata.hosted_by: partner`. An order can target zones at any number of partners, and SRM needs

  somewhere to remember which of them have answered; OEG also only updates an instance it finds in

  the final event's `instances[]`. Because `service_instance.zone_id` is a non-null foreign key into

  SRM's `zone` table, SRM records the partner zone there first, as `kind: federation_business`,

  from FM's `GET /internal/federated-zones/{zone_id}`. `list_resource_zones` only returns

  `kind: resource`, so these rows don't reach placement candidates or `GET /internal/zones` twice,

  and a later deploy to a recorded partner zone still goes to FM. Drift reconciliation must skip

  `hosted_by: partner` instances, since SRM can't observe them; it doesn't exist yet.

- **SRM completes the order from FM's answer.** The order stays `in_progress` while any target is

  `creating`. Each `event.fm.operation.completed` moves its instance to `active` or `failed`; once

  none is left `creating`, SRM completes the order (`completed`, `partially_completed` or `failed`)

  and publishes `event.srm.operation.completed` listing every instance.

FM keeps its `federation_transactions` row unchanged, as the audit and dispute record that ties the

partner's `appInstIdentifier` to our `operation_id`.

### What this amends

- **ADR-0001 and ADR-0022** make SRM the only consumer of `OOP_TASKS`. SRM stays the only consumer

  of `command.srm.>`; FM becomes the consumer of `command.fm.>`.

- **ADR-0046** says the `command.fm.federation.*` family is not created and the DataBus "stays

  scoped to the `command.srm.*` / `event.srm.*` flows FM already uses". The first half stands:

  operator-triggered federation lifecycle remains synchronous HTTP from OEG, and no

  `command.fm.federation.*` subject exists. The second half now also covers the SRM↔FM lifecycle

  hand-off, `command.fm.service.*` and `event.fm.*`.

## Consequences

Easier: SRM and FM share one transport and one set of delivery rules for lifecycle work, and FM's

consumer gets JetStream's redelivery and dead-lettering for free. Nothing holds a connection open

across a partner exchange. Local and federated fulfilment end in the same terminal event, so OEG

needs no federation awareness. Deploy and terminate share a shape.

Harder: `OOP_TASKS` gains a second consumer and `OOP_EVENTS` a second event family, so the stream

definitions in three documents change together. FM becomes a command consumer as well as a

publisher. A bad zone now fails through an event instead of an immediate HTTP error, one hop later.

SRM has to move its publishing onto JetStream, which its `command.srm.*` side needs regardless.

Federated terminate carries a weaker completion guarantee than local terminate, and that difference

is now explicit on the wire.

Revisit when the runtime coordinator described in the platform's future-architecture document

(§6.1) gets built: it's meant to own routing between local and federated fulfilment and would take

over both hops.

---

type: Decision Record

title: "ADR-0054: FM receives every GSMA callback through one router under /operatorplatform/federation/v1/callbacks/{kind}"

description: "FM receives every GSMA callback through one router under /operatorplatform/federation/v1/callbacks/{kind}"

tags: [adr]

timestamp: 2026-09-22T00:00:00Z

status: "proposed"

---

# ADR-0054: FM receives every GSMA callback through one router under `/operatorplatform/federation/v1/callbacks/{kind}`

**Status:** proposed

**Date:** 2026-09-22

**Affects:** [fm/requirements-and-design.md](/fm/requirements-and-design.md) (new §L.1a, callbacks FM receives), [fm/overview-and-architecture.md](/fm/overview-and-architecture.md) (partner authentication)

## Context

When FM federates with a partner as the originating OP, several OPG.04 operations make the partner

call FM back later: `CreateFederation` hands over a `partnerStatusLink`, `InstallApp` an

`appInstCallbackLink`, `ZoneSubscribe` an `availZoneNotifLink`. OPG.04 v6.0 (artifact v1.4.0)

defines eleven such callbacks, and every one has the same shape: a `POST` to a URL the originating

OP chose, authorised with the `notifClientCredentials` scheme and the `fed-mgmt-notif` scope.

FM already sends callbacks as the partner (`onInstanceStatusEvent` after an inbound `InstallApp`),

but it receives none. It hands every partner a `partnerStatusLink` today (`FM_PARTNER_STATUS_LINK`,

default `/operatorplatform/federation/v1/partner-status`) and nothing listens there, so a partner's

`onPartnerStatusEvent` gets a `404`. The deploy hand-off in ADR-0053 depends on FM receiving

`onInstanceStatusEvent` too.

Two more gaps turned up while checking this. REQ-FED-01 says FM validates `fed-mgmt-notif` on

inbound calls, but `PartnerAuthenticator` hardcodes `fed-mgmt`. And no `federation` realm defines a

`fed-mgmt-notif` scope, neither FM's dev realm nor the platform Keycloak chart in helm MR

oop/deploy/helm!12, so a partner can't obtain a callback token from us, and FM's own callback client

can't obtain one from a partner running the same setup.

## Decision

FM receives all GSMA callbacks through one router, and every callback goes through the same three

steps before a per-kind handler runs.

```mermaid

flowchart LR

    P[Partner OP] -->|"POST /operatorplatform/federation/v1/callbacks/{kind}<br/>Bearer token, scope fed-mgmt-notif"| R

    subgraph FM ["FM, originating side"]

        R[Callback router] --> A["Authenticate<br/>scope fed-mgmt-notif<br/>azp to an active partner"]

        A --> C["Resolve context<br/>federationContextId must be an<br/>outbound context of that partner"]

        C --> D{kind}

        D -->|partner-status| H1["onPartnerStatusEvent<br/>federation_contexts.status"]

        D -->|instance-status| H2["onInstanceStatusEvent<br/>federation_transactions"]

        D -->|zone-resource| H3["onZoneResourceUpdateEvent<br/>shared_resource_catalogues"]

    end

```

**URLs.** Callbacks live under `/operatorplatform/federation/v1/callbacks/{kind}`, with one `kind`

per callback: `partner-status`, `instance-status`, `zone-resource`, and more as FM starts calling the

operations that trigger them. OPG.04 leaves the URL to the originating OP; putting it under the

partner-facing prefix means the ingress already routes it to FM, with no new rule. The

`FM_PARTNER_STATUS_LINK` default moves to `/operatorplatform/federation/v1/callbacks/partner-status`.

Nothing received on the old URL, so the move breaks nothing.

**Authentication.** A callback needs a token from our `federation` realm carrying `fed-mgmt-notif`

(REQ-FED-01). A `fed-mgmt` token alone gets `401`: the two scopes separate "you may call our API"

from "you may notify us", which is the distinction OPG.04 draws. The `azp` claim resolves to an

active partner exactly as for other EWBI calls.

**Context check.** Every callback payload carries `federationContextId`. It must name an

**outbound** federation context belonging to the calling partner, meaning one FM created with them.

Anything else gets `404`, so no partner can touch a federation it isn't party to.

**Handlers** are idempotent, because partners may retry, and answer with the status code OPG.04 gives

each callback (`204` for `onPartnerStatusEvent` and `onInstanceStatusEvent`, `200` for

`onZoneResourceUpdateEvent`).

**Order of work.** `onPartnerStatusEvent` comes first: partners can already send it, and its effect

is specified (it updates `federation_contexts.status`, whose values mirror the callback's `Status`

enum). `onInstanceStatusEvent` follows with FM's outbound `InstallApp`, which creates the transaction

it updates, and publishing `event.fm.operation.completed` from it waits on ADR-0053.

`onZoneResourceUpdateEvent` waits for `shared_resource_catalogues`.

**Keycloak.** The `federation` realm defines a `fed-mgmt-notif` client scope and attaches it to each

partner client as an **optional** scope, beside the default `fed-mgmt`. Optional means a token only

carries it when the client asks, so an ordinary API token can't double as a callback token. This

applies to FM's dev realm and to the platform Keycloak chart.

## Consequences

Easier: each new callback is one handler, not a new endpoint with its own auth and ownership checks.

A partner can finally report federation status changes to us. The receiving side of ADR-0053's

deploy flow has a place to live.

Harder: the platform Keycloak chart needs a second client scope per partner client, which its

current template (one `clientScope` value) doesn't support. Partners federating with us have to

request `fed-mgmt-notif` when they call us back, which OPG.04 already requires of them.

* [ADR-0049](0049-terminal-record-read-follows-camara-schema.md) - terminal-record read behaviour follows each family's own vendored CAMARA schema: Traffic Influence returns 200 with `state: deleted`, QoD stays 404 (amends ADR-0034's blanket 404 rule and its 204 delete code)

* [ADR-0050](0050-oeg-federation-lifecycle-proxy-errors-are-gsma-problem-json.md) - OEG's federation-lifecycle proxy endpoints (§N.7, ADR-0046) return OPG.04 `application/problem+json` `ProblemDetails` for every error - FM's errors relayed unchanged, OEG's own 400/401/403/502/503 from a fixed template; ADR-0030's `ErrorInfo` rule is scoped to OEG's CAMARA northbound surface

* [ADR-0051](0051-oeg-fronts-federation-lifecycle-ewbi-byte-for-byte.md) - OEG fronts eight GSMA federation-lifecycle operations on its own platform namespace, `/platform/v1/federation/partners/{partnerOpId}/federations/...`, scoped by a registered `partnerOpId` (create/read/update/delete federation, fed-context-id lookup, zone list/subscribe/details/unsubscribe), forwarding the raw GSMA request/response byte-for-byte to FM including the query string, pinned to GSMA OPG.04 v6.0 / artifact v1.4.0; operator-triggered outbound only, separated from inbound EWBI at the ingress; keeps ADR-0046's sync-HTTP/stateless/no-DataBus transport and ADR-0050's error model (amends this ADR's own first version, which used the bare canonical OPG.04 paths and lost the partner addressing GSMA gets for free from the URL)

* [ADR-0053](0053-srm-fm-federated-lifecycle-handoff-over-the-databus.md) - SRM hands federated deploys and terminates to FM as `command.fm.service.deploy` / `.terminate` on `OOP_TASKS`, and FM reports the outcome as `event.fm.operation.completed` on `OOP_EVENTS`; zone lookups stay synchronous HTTP reads. `completion_basis` separates a confirmed deploy from a terminate the partner only accepted. SRM writes a `service_order` row and a `service_instance` row per partner target (`hosted_by: partner`, partner zone recorded as `federation_business`) and completes the order from FM's answer; FM waits 30 minutes by default for a deploy callback. Amends ADR-0001/0022 (FM consumes `command.fm.>`) and ADR-0046's DataBus scope; resolves G4

* [ADR-0054](0054-fm-receives-gsma-callbacks-through-one-router.md) - FM receives every OPG.04 callback on `/operatorplatform/federation/v1/callbacks/{kind}`: one router that checks the `fed-mgmt-notif` scope and that `federationContextId` is an outbound context of the calling partner, then a per-kind idempotent handler. `onPartnerStatusEvent` first; `FM_PARTNER_STATUS_LINK` moves to `/callbacks/partner-status`. The `federation` realm gets `fed-mgmt-notif` as an optional scope on partner clients

* [ADR-0061](0061-fm-onboards-apps-at-partners-over-ewbi-before-installapp.md) - FM onboards an application at a partner over the EWBI (`UploadFile`, `UploadArtefact`, `OnboardApplication`) on the first `InstallApp` of that specification version, with derived GSMA ids, public repositories and container units only for now, and `flavourId` from the zone subscription; the partner creates the specification in its SRM under the originating partner as tenant and resolves `InstallApp` from that onboarding, replacing ADR-0018's manual app mapping. Adds `onApplicationOnboardStatusEvent` to ADR-0054's router