docs(fm,srm): SRM→FM federated hand-off over the DataBus (ADR-0053)

This MR settles how SRM hands a federated deploy or terminate to FM and how the outcome gets back. ADR-0044 gave SRM the "is this zone ours or a partner's?" decision but never said how SRM passes the work on, which is what open item G4 in the FM overview doc was waiting for. G4 is closed here.

Deploy

sequenceDiagram
    autonumber
    actor AP as App Provider
    box Our operator
        participant OEG
        participant SRMA as SRM-A
        participant FMA as FM-A
    end
    box Partner operator
        participant FMB as FM-B
        participant SRMB as SRM-B
    end
    AP->>OEG: POST /appinstances (CAMARA)
    OEG-->>AP: 202 appInstanceId
    OEG->>SRMA: command.srm.service.deploy [OOP_TASKS]
    SRMA->>FMA: GET /internal/federated-zones/{zone_id}
    FMA-->>SRMA: 200 zone is federated
    Note over SRMA: writes service_order row
    SRMA->>FMA: command.fm.service.deploy [OOP_TASKS]
    Note over FMA: writes federation_transactions row
    FMA->>FMB: InstallApp (GSMA EWBI)
    FMB-->>FMA: 202 appInstIdentifier
    FMB->>SRMB: command.srm.service.deploy [OOP_TASKS]
    SRMB->>FMB: event.srm.operation.completed [OOP_EVENTS]
    FMB->>FMA: onInstanceStatusEvent (GSMA callback)
    FMA->>SRMA: event.fm.operation.completed, partner_confirmed [OOP_EVENTS]
    SRMA->>OEG: event.srm.operation.completed [OOP_EVENTS]
    OEG->>AP: callback, or App Provider polls

The App Provider gets their 202 from OEG at step 2, before SRM has even read the command. So nothing between SRM and FM needs a synchronous answer. The HTTP 202 FM would have sent SRM only said "I took the job", and the JetStream acknowledgement on command.fm.service.deploy says the same. The outcome can't ride on the request anyway, because GSMA's InstallApp only ever answers 202 and reports the real result later through a callback. And FM can't call SRM back over HTTP, since SRM's contract rules out HTTP calls that change lifecycle state.

The zone lookup (steps 4 and 5) stays HTTP bc it's a sync call. A GET /internal/federated-zones list variant is added too, since SRM's zone listing already calls it.

Terminate

sequenceDiagram
    autonumber
    actor AP as App Provider
    box Our operator
        participant OEG
        participant SRMA as SRM-A
        participant FMA as FM-A
    end
    box Partner operator
        participant FMB as FM-B
        participant SRMB as SRM-B
    end
    AP->>OEG: DELETE /appinstances/{id} (CAMARA)
    Note over OEG: operations row PENDING, app_instances row TERMINATING
    OEG->>SRMA: command.srm.service.terminate [OOP_TASKS]
    OEG-->>AP: 202
    Note over SRMA: instance id not in local inventory, so federated
    Note over SRMA: writes service_order row
    SRMA->>FMA: command.fm.service.terminate [OOP_TASKS]
    Note over FMA: resolves the instance from federation_transactions
    FMA->>FMB: RemoveApp (GSMA EWBI)
    FMB->>SRMB: command.srm.service.terminate [OOP_TASKS]
    FMB-->>FMA: 200 termination accepted
    FMA->>SRMA: event.fm.operation.completed, partner_accepted
    SRMB->>FMB: event.srm.operation.completed [OOP_EVENTS]
    SRMA->>OEG: event.srm.operation.completed with instances[] [OOP_EVENTS]
    Note over OEG: app_instances row TERMINATED
    OEG->>AP: callback, or App Provider polls

GSMA gives RemoveApp no callback and no terminal state to poll for, so FM-A reports as soon as the partner says yes (step 11), before the partner's SRM has actually removed anything. The event carries completion_basis so nobody reads more into it than that:

completion_basis Operation Means
partner_confirmed deploy the partner's callback says the instance is running
partner_accepted terminate the partner accepted; removal isn't confirmed
deadline_expired deploy no callback within 30 minutes, reported as failed

What has to change on the DataBus

This is where most of the review attention should go:

  • OOP_TASKS gains command.fm.> with an fm-task-worker consumer next to SRM's. That amends ADR-0001 and ADR-0022, which make SRM its only consumer. WorkQueue streams allow it because the two filters don't overlap.
  • OOP_EVENTS gains event.fm.>. Today it only captures event.srm.>, in the docs and in FM's code, so without this change the result event wouldn't be stored at all.
  • ADR-0046's line that the DataBus "stays scoped to command.srm.* / event.srm.*" gets a carve-out for this hand-off. Its actual decision, operator-triggered federation lifecycle over HTTP from OEG with no command.fm.federation.*, is untouched. I didn't edit ADR-0046 itself because it lives in !3 (merged); ADR-0 053 states the amendment instead.
  • ⚠️ This is an important one! ⚠️ Every publisher, SRM included, publishes through JetStream and waits for the acknowledgement. SRM's refactor/fm-client publishes with plain NATS right now, so a command sent while FM restarts would vanish with no error.

Also decided

Refusals and retries. If FM can't act (no partner has the zone, the agreement forbids it, the instance is unknown), it answers with a failed event.fm.operation.completed carrying the reason. An unreachable partner gets a negative acknowledgement and JetStream redelivers; FM reports the failure once deliveries run out. operation_id is the idempotency key throughout.

Deadline. FM waits 30 minutes by default for the deploy callback, counted from the transaction's stored start time so a restart doesn't lose it. A cold edge node pulling a multi-gigabyte AI image can take many minutes, and a deadline that fired during a normal slow deploy would report a running app as failed.

Files

File Change
decisions/0053-srm-fm-federated-lifecycle-handoff-over-the-databus.md the decision record (renamed from the first version's …-split-request-and-result.md)
decisions/index.md index entry
fm/requirements-and-design.md SRM→FM hand-off section rewritten: zone reads, the commands FM consumes, refusal types; FM's DataBus sections and REQ-FED-21 updated; FmOperationCompletedV1 schema
srm/interface-contract.md stream config, subject table and the federated hand-off flow
srm/architecture.md UC-SRM-03a updated, open item F6 closed
platform/architecture.md DataBus section: both streams, consumers, the communication diagram
fm/overview-and-architecture.md UC-FM-07 flow updated, open item G4 closed

Before merging

This depends on !3 (merged) (oeg-fm): the contract refers to ADR-0046, ADR-0050 and ADR-0051, which only exist there. The MR targets fm-persistence-model so the diff shows only this work, and it'll need retargeting to develop once the branches underneath have merged.

0052 is skipped on purpose. oeg-fm has renumbered its ADRs once, and the gap keeps a second renumber there from colliding with this one.

Most of the implementation lands on SRM (publishing command.fm.* through JetStream, consuming event.fm.operation.completed, the service_order row, instances[] in the completion), so stentoumis's read of the SRM contract changes matters most.

Edited by Sergio Gimenez

Merge request reports

Loading
Loading