docs(fm,srm): SRM→FM federated hand-off over the DataBus (ADR-0053)
This MR settles how SRM hands a federated deploy or terminate to FM and how the outcome gets back. ADR-0044 gave SRM the "is this zone ours or a partner's?" decision but never said how SRM passes the work on, which is what open item G4 in the FM overview doc was waiting for. G4 is closed here.
Deploy
sequenceDiagram
autonumber
actor AP as App Provider
box Our operator
participant OEG
participant SRMA as SRM-A
participant FMA as FM-A
end
box Partner operator
participant FMB as FM-B
participant SRMB as SRM-B
end
AP->>OEG: POST /appinstances (CAMARA)
OEG-->>AP: 202 appInstanceId
OEG->>SRMA: command.srm.service.deploy [OOP_TASKS]
SRMA->>FMA: GET /internal/federated-zones/{zone_id}
FMA-->>SRMA: 200 zone is federated
Note over SRMA: writes service_order row
SRMA->>FMA: command.fm.service.deploy [OOP_TASKS]
Note over FMA: writes federation_transactions row
FMA->>FMB: InstallApp (GSMA EWBI)
FMB-->>FMA: 202 appInstIdentifier
FMB->>SRMB: command.srm.service.deploy [OOP_TASKS]
SRMB->>FMB: event.srm.operation.completed [OOP_EVENTS]
FMB->>FMA: onInstanceStatusEvent (GSMA callback)
FMA->>SRMA: event.fm.operation.completed, partner_confirmed [OOP_EVENTS]
SRMA->>OEG: event.srm.operation.completed [OOP_EVENTS]
OEG->>AP: callback, or App Provider pollsThe App Provider gets their 202 from OEG at step 2, before SRM has even read the command. So nothing between SRM and FM needs a synchronous answer. The HTTP 202 FM would have sent SRM only said "I took the job", and the JetStream acknowledgement on command.fm.service.deploy says the same. The outcome can't ride on the request anyway, because GSMA's InstallApp only ever answers 202 and reports the real result later through a callback. And FM can't call SRM back over HTTP, since SRM's contract rules out HTTP calls that change lifecycle state.
The zone lookup (steps 4 and 5) stays HTTP bc it's a sync call. A GET /internal/federated-zones list variant is added too, since SRM's zone listing already calls it.
Terminate
sequenceDiagram
autonumber
actor AP as App Provider
box Our operator
participant OEG
participant SRMA as SRM-A
participant FMA as FM-A
end
box Partner operator
participant FMB as FM-B
participant SRMB as SRM-B
end
AP->>OEG: DELETE /appinstances/{id} (CAMARA)
Note over OEG: operations row PENDING, app_instances row TERMINATING
OEG->>SRMA: command.srm.service.terminate [OOP_TASKS]
OEG-->>AP: 202
Note over SRMA: instance id not in local inventory, so federated
Note over SRMA: writes service_order row
SRMA->>FMA: command.fm.service.terminate [OOP_TASKS]
Note over FMA: resolves the instance from federation_transactions
FMA->>FMB: RemoveApp (GSMA EWBI)
FMB->>SRMB: command.srm.service.terminate [OOP_TASKS]
FMB-->>FMA: 200 termination accepted
FMA->>SRMA: event.fm.operation.completed, partner_accepted
SRMB->>FMB: event.srm.operation.completed [OOP_EVENTS]
SRMA->>OEG: event.srm.operation.completed with instances[] [OOP_EVENTS]
Note over OEG: app_instances row TERMINATED
OEG->>AP: callback, or App Provider pollsGSMA gives RemoveApp no callback and no terminal state to poll for, so FM-A reports as soon as the partner says yes (step 11), before the partner's SRM has actually removed anything. The event carries completion_basis so nobody reads more into it than that:
completion_basis |
Operation | Means |
|---|---|---|
partner_confirmed |
deploy | the partner's callback says the instance is running |
partner_accepted |
terminate | the partner accepted; removal isn't confirmed |
deadline_expired |
deploy | no callback within 30 minutes, reported as failed |
What has to change on the DataBus
This is where most of the review attention should go:
OOP_TASKSgainscommand.fm.>with anfm-task-workerconsumer next to SRM's. That amends ADR-0001 and ADR-0022, which make SRM its only consumer. WorkQueue streams allow it because the two filters don't overlap.OOP_EVENTSgainsevent.fm.>. Today it only capturesevent.srm.>, in the docs and in FM's code, so without this change the result event wouldn't be stored at all.- ADR-0046's line that the DataBus "stays scoped to
command.srm.*/event.srm.*" gets a carve-out for this hand-off. Its actual decision, operator-triggered federation lifecycle over HTTP from OEG with nocommand.fm.federation.*, is untouched. I didn't edit ADR-0046 itself because it lives in !3 (merged); ADR-0 053 states the amendment instead. ⚠️ This is an important one!⚠️ Every publisher, SRM included, publishes through JetStream and waits for the acknowledgement. SRM'srefactor/fm-clientpublishes with plain NATS right now, so a command sent while FM restarts would vanish with no error.
Also decided
Refusals and retries. If FM can't act (no partner has the zone, the agreement forbids it, the instance is unknown), it answers with a failed event.fm.operation.completed carrying the reason. An unreachable partner gets a negative acknowledgement and JetStream redelivers; FM reports the failure once deliveries run out. operation_id is the idempotency key throughout.
Deadline. FM waits 30 minutes by default for the deploy callback, counted from the transaction's stored start time so a restart doesn't lose it. A cold edge node pulling a multi-gigabyte AI image can take many minutes, and a deadline that fired during a normal slow deploy would report a running app as failed.
Files
| File | Change |
|---|---|
decisions/0053-srm-fm-federated-lifecycle-handoff-over-the-databus.md |
the decision record (renamed from the first version's …-split-request-and-result.md) |
decisions/index.md |
index entry |
fm/requirements-and-design.md |
SRM→FM hand-off section rewritten: zone reads, the commands FM consumes, refusal types; FM's DataBus sections and REQ-FED-21 updated; FmOperationCompletedV1 schema |
srm/interface-contract.md |
stream config, subject table and the federated hand-off flow |
srm/architecture.md |
UC-SRM-03a updated, open item F6 closed |
platform/architecture.md |
DataBus section: both streams, consumers, the communication diagram |
fm/overview-and-architecture.md |
UC-FM-07 flow updated, open item G4 closed |
Before merging
This depends on !3 (merged) (oeg-fm): the contract refers to ADR-0046, ADR-0050 and ADR-0051, which only exist there. The MR targets fm-persistence-model so the diff shows only this work, and it'll need retargeting to develop once the branches underneath have merged.
0052 is skipped on purpose. oeg-fm has renumbered its ADRs once, and the gap keeps a second renumber there from colliding with this one.
Most of the implementation lands on SRM (publishing command.fm.* through JetStream, consuming event.fm.operation.completed, the service_order row, instances[] in the completion), so stentoumis's read of the SRM contract changes matters most.