docs(fm,srm): move the SRM->FM hand-off request onto the DataBus (ADR-0053)

---

type: Decision Record

title: "ADR-0053: The SRM→FM federated lifecycle hand-off is a synchronous internal HTTP request plus an asynchronous DataBus result event"

description: "The SRM→FM federated lifecycle hand-off is a synchronous internal HTTP request plus an asynchronous DataBus result event"

tags: [adr]

timestamp: 2026-09-22T00:00:00Z

status: "proposed"

---

# ADR-0053: The SRM→FM federated lifecycle hand-off is a synchronous internal HTTP request plus an asynchronous DataBus result event

**Status:** proposed

**Date:** 2026-09-22

**Affects:** [fm/overview-and-architecture.md](/fm/overview-and-architecture.md) (§G4 open item, UC-FM-07), [fm/requirements-and-design.md](/fm/requirements-and-design.md) (§L.3a, the SRM→FM internal HTTP surface), [srm/interface-contract.md](/srm/interface-contract.md) (§C.4 events, §F.4a the federated hand-off flow), [platform/architecture.md](/platform/architecture.md) (Component Boundary Matrix)

> **Numbering note.** `0053` clears the whole range in flight on the `oeg-fm` branch

> (`0046`, `0049`, `0050`, `0051`) and `0048` on `docs/platform-roles`, deliberately skipping

> `0052` so a further renumber on either branch cannot collide with this record. `oeg-fm` has

> already renumbered once (`0047`/`0048` → `0050`/`0051`). Re-check at MR time.

## Context

ADR-0044 — the decision that moved "is this deploy target zone local or federated?" from OEG to

SRM — established that SRM resolves zone locality and, when the zone is not in its own inventory,

hands the work to FM. The 2026-09-01 TSC+MDL meeting confirmed that ownership but did not conclude

how the hand-off is carried. That gap is recorded as open item §G4 in the FM overview document,

whose standing recommendation is "synchronous internal HTTP for consistency with the existing FM↔SRM

internal-call pattern". The SRM interface contract's federated hand-off flow (§F.4a) and FM's

SRM→FM internal HTTP surface (§L.3a) both carry the question forward, §L.3a stating its shapes are

"indicative, not final".

Four facts settle the shape, and none of them were on the table when §G4 was written.

**The deploy result cannot travel on the request hop.** The GSMA OPG.04 v6.0 machine contract

(`OPG.04_EWBI_Federation_API_v1.4.0.yaml`) defines `InstallApp` with **`202` as its only success

response** plus an `onInstanceStatusEvent` callback delivered to the `appInstCallbackLink` the

request must carry (a required body field). The partner returns no instance outcome on the HTTP

response; it arrives later as a partner-initiated callback secured with the

`notifClientCredentials` scheme and the `fed-mgmt-notif` scope. FM therefore has nothing to return

to SRM but "accepted", however the hand-off is carried. A second channel for the eventual result is

required in every variant, so the transport question is really two questions, one per hop.

**Termination has no completion signal at all.** The same contract defines `RemoveApp` with `200`

("Application instance termination request accepted") and **no callbacks whatsoever**.

`GetAppInstanceDetails` reports `appInstanceState` from the enum `PENDING | READY | FAILED |

TERMINATING` — which has no terminal value for a completed removal. So neither push nor polling can

observe a federated termination completing. This asymmetry is a property of OPG.04, not a choice

available to us.

**FM cannot call SRM back over HTTP.** The SRM interface contract's boundary rule (§A) states: "SRM

exposes **no** public HTTP lifecycle mutation endpoints. Internal HTTP is for reads, diagnostics,

trusted catalog/admin management, and synchronous query/data calls only." Completing a deploy or

terminate operation is a lifecycle mutation, so a completion endpoint on SRM would contradict SRM's

own rule.

**Nobody currently records a federated operation.** The platform architecture's Component Boundary

Matrix assigns "Operation lifecycle state management" to "SRM (service orders, instances) + OEG

(operations)" and names FM explicitly as *not* an owner. FM's own persistence model agrees: its

entity-ownership table records "Service instance detail" as not FM-owned, because "SRM owns the

instance record; FM keeps only `operation_id`/result summary in a transaction", under a tie-breaker

that FM holds only UUID references to anything SRM owns. But the SRM hand-off flow (§F.4a step 4)

says "No local `service_order`/`service_instance` rows are written for the federated portion."

Between them, a federated deploy is recorded by no component — leaving no idempotency anchor and no

way for SRM to correlate a late result after a restart.

Options considered for the transport:

- Option A: synchronous internal HTTP for both hops, FM calling back into SRM to complete the

  operation. Rejected: contradicts SRM's no-lifecycle-mutation boundary, and adds an SRM HTTP

  surface that exists for nothing else.

- Option B: a new async `command.fm.*` family on `OOP_TASKS` for the request hop plus an event for

  the result. Rejected for the request hop: ADR-0046 — the decision that moved operator-triggered

  federation lifecycle from the DataBus to synchronous OEG→FM HTTP — explicitly declined to create

  `command.fm.federation.*`, stating "the DataBus stays scoped to the `command.srm.*` /

  `event.srm.*` flows FM already uses". The request hop *is* answerable in one round trip, and

  SRM's own JetStream redelivery already supplies durability one layer up.

- Option C: synchronous internal HTTP for the request hop, DataBus event for the result hop.

- Option D: SRM polls FM for the outcome. Rejected: converts a push into a poll across a component

  boundary, and OPG.04 bounds neither callback latency nor termination duration.

## Decision

**Option C, applied to both deploy and terminate.** Each hop uses the transport its nature calls

for, and the two lifecycle operations share one shape.

### Hop 1 — request: synchronous internal HTTP

SRM calls FM and receives an immediate acknowledgement, before any partner exchange begins:

| Operation | Endpoint | Payload |

|---|---|---|

| Federated deploy | `POST /internal/federation/deploy` | `SrmServiceDeployV1`, unshaped |

| Federated terminate | `POST /internal/federation/terminate` | `SrmServiceTerminateV1`, unshaped |

SRM performs no GSMA/EWBI translation in either case — FM alone owns the mapping to `InstallApp`

and `RemoveApp` (ADR-0044). FM validates, writes its `federation_transactions` audit row, and

answers `202` with the accepted handle.

This keeps the direction consistent with FM's other internal HTTP surfaces — the OEG→FM

subscriber-routing hand-off (§L.3), the OEG→FM federation-lifecycle proxy established by ADR-0051

(§L.3b), and FM's reads into SRM (§L.4) — and creates no new DataBus subject family, honouring

ADR-0046.

The lookups that precede a hand-off stay synchronous internal HTTP, since each is answerable in one

round trip:

- **Deploy:** `GET /internal/federated-zones/{zone_id}`, already specified in §L.3a. A `404` means

  the zone is unknown anywhere and is treated as an ordinary placement failure.

- **Terminate:** no separate lookup. SRM finds the `service_instance_id` absent from its local

  inventory and posts the terminate to FM, which resolves the instance from its own

  `federation_transactions` records. A `404` from FM means the instance is unknown to both

  components and is an ordinary not-found failure.

### Hop 2 — result: `event.fm.operation.completed` on `OOP_EVENTS`

FM publishes one terminal event per `operation_id`, carrying the originating `operation_id` and the

outcome. SRM consumes it and completes the original command's lifecycle by publishing

`event.srm.operation.completed` exactly as it would for local fulfillment. OEG — and through it the

App Provider — sees one uniform terminal event and never learns where fulfillment happened, which is

also what topology hiding requires.

The event fires at different moments for the two operations, because OPG.04 gives them different

guarantees:

| Operation | FM publishes the result when | Meaning of `completed` |

|---|---|---|

| Deploy | the partner's `onInstanceStatusEvent` callback arrives, or FM's callback deadline expires | the partner reports the instance running |

| Terminate | the partner answers `RemoveApp` with `200` | **the partner accepted the termination** — not that removal finished |

The terminate semantics are weaker deliberately. OPG.04 defines no callback for `RemoveApp` and no

`TERMINATED` value in `InstanceState`, so no stronger claim is observable. The event carries

`metadata.completion_basis` of `partner_confirmed` (deploy) or `partner_accepted` (terminate) so

consumers can tell the two apart rather than inferring a guarantee that does not exist. Detecting a

termination the partner accepted but never performed is left to reconciliation, not to this hop.

The weaker guarantee is acceptable because the incentives point the right way: a partner hosting an

application it was asked to remove is spending its own compute, storage and power on a workload it

cannot bill for once the federated agreement's usage record closes. The party best placed to perform

the cleanup is also the party that wants it done. This is a reason not to build enforcement

machinery for a case the counterparty is already motivated to avoid — not a guarantee, which is why

reconciliation remains the backstop.

### FM's deploy callback deadline

FM waits **30 minutes** by default for a partner's `onInstanceStatusEvent` before declaring the

deploy failed. The value is configuration, not contract: operators with slower partners raise it,

and a federation agreement may later carry a per-partner override.

30 minutes is deliberately generous. A cold edge node pulling a multi-gigabyte image — routine for

AI and GPU workloads — can take many minutes before the instance is schedulable, and the partner may

queue the request behind others. A deadline tight enough to fire during normal operation would

convert a slow success into a reported failure, which is worse than waiting, because the application

is running on the partner's infrastructure either way and we would have told the App Provider

otherwise.

Two properties this deadline must have:

- **It is derived from the stored `federation_transactions.started_at`, not an in-process timer**,

  so it survives an FM restart. An in-memory timer would leave a restarted FM waiting forever on a

  callback whose deadline it has forgotten.

- **Expiry is distinguishable from partner-reported failure.** FM publishes the terminal event with

  a `metadata.completion_basis` of `deadline_expired` rather than `partner_confirmed`, so an

  operator can tell "the partner said it failed" from "the partner never answered" — different

  problems with different remedies.

This introduces the `event.fm.*` namespace. It does **not** reopen ADR-0046: no `command.fm.*`

family is created, nothing ADR-0046 or ADR-0051 routes over HTTP moves to the DataBus, and the new

subject exists only for outcomes that have no synchronous answer. ADR-0022 — the decision adopting

NATS JetStream — already states "`OOP_EVENTS` is not restricted to SRM: non-SRM event families

share the stream", so this uses the stream as designed rather than widening it.

### Record ownership — amending the SRM hand-off flow (§F.4a step 4)

The sentence "No local `service_order`/`service_instance` rows are written for the federated

portion" splits:

- **SRM writes a `service_order` row** for federated deploys and terminates, as it does for every

  command it accepts. This is the idempotency anchor via its `UNIQUE(operation_id)` constraint, and

  the record that lets SRM correlate a late `event.fm.operation.completed` after a restart. The

  platform's Component Boundary Matrix already assigns service orders to SRM. The row carries the

  federated outcome on completion.

- **SRM writes no `service_instance` rows.** This half of §F.4a stands. A federated instance runs on

  the partner's infrastructure; SRM's inventory exists to serve placement, best-effort capacity

  queries and drift reconciliation, and an instance it cannot see would produce permanent false

  drift. It is also structurally impossible today: `service_instance.zone_id` is a not-null foreign

  key into SRM's `zone` table, and a federated zone is by definition absent from it.

- **FM keeps its `federation_transactions` row** unchanged — the audit and dispute-resolution

  record, correlating the partner's external handle with our internal `operation_id`, exactly as

  FM's persistence model already specifies.

The caller-facing record is unaffected: OEG pre-creates and maintains its own `app_instances` rows

and answers the App Provider's reads from `oeg_db`, so nothing depends on SRM holding an instance

row for a federated deploy.

### Retry, timeout and idempotency

SRM owns hop 1. It retries the `POST` within its own already-async `command.srm.*` handling, where

JetStream redelivery plus the `service_order.UNIQUE(operation_id)` row this decision restores

provide at-least-once with idempotency (§D.4 of the SRM interface contract, its delivery-guarantee

section).

FM owns hop 2 and everything beyond it: the partner handshake, the callback wait and its deadline,

and publishing the terminal event exactly once per `operation_id`. Neither OEG nor the original

caller participates in either hop.

`operation_id` is the idempotency key on both hops. A repeated request carrying an `operation_id`

FM has already accepted returns `202` with the existing handle and starts no second EWBI exchange.

## Consequences

Easier: each hop uses the transport matching its nature, so no connection and no operation is held

open across a partner WAN exchange. SRM needs no new HTTP surface, so its boundary rule stands. FM's

retry, deadline and callback handling stay inside FM, where the partner relationship already lives.

Federated and local fulfillment converge on the same terminal event, so OEG needs no awareness of

which occurred. Deploy and terminate share one hand-off shape, so neither needs a separate mechanism.

Harder: the platform gains one event family outside `event.srm.*`, and readers must not

over-generalise it into permission for `command.fm.*` — ADR-0046 still governs that. FM becomes an

`OOP_EVENTS` publisher as well as a consumer, growing its JetStream wiring. SRM consumes an event

family it does not own, which is new for SRM. Federated terminate carries a weaker completion

guarantee than local terminate, and that difference is now explicit on the wire rather than hidden.

One follow-up is left open deliberately. **Reconciliation of accepted-but-unperformed terminations**

has no mechanism here; it belongs with drift detection, and SRM's existing

`event.srm.inventory.drift_detected` family is the natural home once federated instances are visible

to it. The incentive argument above is why this is a backstop rather than a prerequisite.

Revisit when the "runtime coordinator" component described in the platform's future-architecture

document (§6.1) is built: it is intended to own cross-domain routing between local and federated

fulfillment, and would absorb both hops.

* [ADR-0049](0049-terminal-record-read-follows-camara-schema.md) - terminal-record read behaviour follows each family's own vendored CAMARA schema: Traffic Influence returns 200 with `state: deleted`, QoD stays 404 (amends ADR-0034's blanket 404 rule and its 204 delete code)

* [ADR-0050](0050-oeg-federation-lifecycle-proxy-errors-are-gsma-problem-json.md) - OEG's federation-lifecycle proxy endpoints (§N.7, ADR-0046) return OPG.04 `application/problem+json` `ProblemDetails` for every error - FM's errors relayed unchanged, OEG's own 400/401/403/502/503 from a fixed template; ADR-0030's `ErrorInfo` rule is scoped to OEG's CAMARA northbound surface

* [ADR-0051](0051-oeg-fronts-federation-lifecycle-ewbi-byte-for-byte.md) - OEG fronts eight GSMA federation-lifecycle operations on its own platform namespace, `/platform/v1/federation/partners/{partnerOpId}/federations/...`, scoped by a registered `partnerOpId` (create/read/update/delete federation, fed-context-id lookup, zone list/subscribe/details/unsubscribe), forwarding the raw GSMA request/response byte-for-byte to FM including the query string, pinned to GSMA OPG.04 v6.0 / artifact v1.4.0; operator-triggered outbound only, separated from inbound EWBI at the ingress; keeps ADR-0046's sync-HTTP/stateless/no-DataBus transport and ADR-0050's error model (amends this ADR's own first version, which used the bare canonical OPG.04 paths and lost the partner addressing GSMA gets for free from the URL)

* [ADR-0053](0053-srm-fm-federated-deploy-handoff-split-request-and-result.md) - the SRM→FM federated lifecycle hand-off splits by hop: a synchronous internal HTTP `POST` (`/internal/federation/deploy`, `/internal/federation/terminate`) answered `202`, then `event.fm.operation.completed` on OOP_EVENTS for the outcome, because OPG.04 returns no result on the request and SRM exposes no HTTP lifecycle mutation endpoint; covers terminate as well as deploy, restores SRM's `service_order` row as the idempotency anchor while keeping no `service_instance` rows, sets a 30-minute default deploy-callback deadline (resolves §G4)

* [ADR-0053](0053-srm-fm-federated-lifecycle-handoff-over-the-databus.md) - SRM hands federated deploys and terminates to FM as `command.fm.service.deploy` / `.terminate` on `OOP_TASKS`, and FM reports the outcome as `event.fm.operation.completed` on `OOP_EVENTS`; zone lookups stay synchronous HTTP reads. `completion_basis` separates a confirmed deploy from a terminate the partner only accepted. SRM writes a `service_order` row for federated work but no `service_instance` row; FM waits 30 minutes by default for a deploy callback. Amends ADR-0001/0022 (FM consumes `command.fm.>`) and ADR-0046's DataBus scope; resolves G4

  1. OEG publishes `command.srm.service.deploy` unconditionally, as it does for every deploy (no locality check in OEG).

  2. SRM checks its own zone inventory; not found locally, so it queries FM for federated zone/capability data.

  3. FM returns the matching federated zone entry (from `shared_resource_catalogues`).

  4. SRM forwards the deploy to FM using FM's internal schema (not raw GSMA/OPG.04) via a synchronous `POST /internal/federation/deploy` answered `202`; the outcome returns later on `event.fm.operation.completed` (ADR-0053, §G4).

  4. SRM writes its `service_order` row and publishes `command.fm.service.deploy` on `OOP_TASKS`, carrying the deploy command it received unchanged (not raw GSMA/OPG.04); the outcome comes back later on `event.fm.operation.completed` (ADR-0053, §G4).

  5. FM maps the request to OPG.04 `InstallApp` and executes the outbound EWBI call to the partner's FM, through the same code path used for inbound federation-originated deploys (UC-FM-01's fulfillment pipeline).

  6. FM reports the result back to SRM, which resumes its normal `event.srm.operation.completed` lifecycle for the original command.

  6. FM publishes `event.fm.operation.completed` on `OOP_EVENTS`; SRM consumes it and publishes `event.srm.operation.completed` for the original command, as for a local deploy.

- **Why this exists:** Keeps placement (local-vs-federated) a single SRM-owned decision instead of splitting it between OEG and FM; see ADR-0044.

- **Interim status:** this SRM-side hand-off is a deliberate stopgap pending a future "runtime coordinator" component (platform/future-architecture.md §6.1) that will eventually own it.

### G4. SRM → FM Hand-off for Federated Deployment (resolved by ADR-0053)

**The question was:** When SRM determines a deploy target zone is federated (UC-FM-07), what is the exact hand-off contract to FM — synchronous internal HTTP or a new async DataBus command?

**The question was:** When SRM finds a deploy's target zone is federated (UC-FM-07), how does it hand the work to FM: synchronous internal HTTP, or an async DataBus command?

**Status: resolved (ADR-0053).** Both, split by hop, and extended to cover termination as well as deployment. The request hop is a synchronous internal HTTP `POST` to FM (`/internal/federation/deploy`, `/internal/federation/terminate`) answered `202`; the result hop is `event.fm.operation.completed` on `OOP_EVENTS`, which SRM consumes and republishes as `event.srm.operation.completed`. A second channel is unavoidable because OPG.04 `InstallApp` returns only `202` plus a later callback, and `RemoveApp` returns `200` with no callback at all; and FM cannot call back into SRM over HTTP because SRM exposes no HTTP lifecycle mutation endpoints. ADR-0053 also amends the SRM hand-off flow so SRM writes a `service_order` row (the idempotency anchor) while still writing no `service_instance` rows, and sets FM's deploy-callback deadline at a 30-minute default. See [srm/interface-contract.md §F.4a](/srm/interface-contract.md) and [fm/requirements-and-design.md §L.3a](/fm/requirements-and-design.md).

**Status: resolved (ADR-0053).** Over the DataBus in both directions, for terminate as well as deploy. SRM publishes `command.fm.service.deploy` or `command.fm.service.terminate` on `OOP_TASKS`, and FM answers with `event.fm.operation.completed` on `OOP_EVENTS`, which SRM turns into its usual `event.srm.operation.completed`. Zone lookups (`GET /internal/federated-zones`) stay synchronous HTTP reads. The earlier recommendation here, HTTP for the request, was dropped in review: OEG has already answered the App Provider before SRM sees the command, so an HTTP `202` from FM would tell SRM nothing a JetStream acknowledgement doesn't, and one hand-off shouldn't need two transports. ADR-0053 also has SRM write a `service_order` row for federated work (no `service_instance` row) and sets FM's deploy-callback wait at 30 minutes by default. See [srm/interface-contract.md §F.4a](/srm/interface-contract.md) and [fm/requirements-and-design.md §L.3a](/fm/requirements-and-design.md).

---