docs(fm): ADR-0067, partner client secret typed in through the registry, stored encrypted

---

type: Decision Record

title: "ADR-0067: FM stores a partner client secret typed in through the registry, AES-GCM encrypted in fm_db; the mounted-file reference stays as the other source"

description: "The operator can register a partner's client secret by value. FM encrypts it with AES-256-GCM under one app-wide key into a partner_op_secrets table, never returns it, and reports only our_client_secret_set. our_client_secret_ref keeps working; a partner has one source or the other. No key configured means 501 for requests that carry a secret."

tags: [adr]

timestamp: 2026-10-13T00:00:00Z

status: "proposed"

---

# ADR-0067: Partner client secrets typed in through the registry are stored encrypted in `fm_db`

**Status:** proposed

**Date:** 2026-10-13

**Tracked in:** federation-manager#28

**Affects:** [fm/overview-and-architecture.md](/fm/overview-and-architecture.md), [fm/implementation-and-delivery.md](/fm/implementation-and-delivery.md), [fm/requirements-and-design.md](/fm/requirements-and-design.md); follows the pattern of [ADR-0063](/decisions/0063-callback-tokens-encrypted-in-oeg-db.md) (OEG stores client callback tokens AES-GCM encrypted under one app-wide key)

## Context

For outbound EWBI calls FM needs the client secret the partner issued to us (OPG.04 Table 371,

the credentials operators exchange before federating). The FM docs say that secret lives only in

a mounted file named by `our_client_secret_ref`, never in `fm_db`, and FM #21 made it a

done-criterion: "no secret value is accepted, stored, returned or logged".

That works for Helm and bootstrap deployments. It doesn't work for the Portal, where the operator

registers a partner from a browser and has no way to put a file inside FM's container. The Portal

form has a "Secret file path" field today, which only someone with shell access to the cluster

can fill in.

Options:

- (a) keep files only, and have the Portal tell the operator to ask ops to mount one;

- (b) a platform secret store (Vault, or Kubernetes Secrets written at runtime): new

  infrastructure for one value per partner;

- (c) encrypt the secret into `fm_db` with one key from FM configuration, as OEG already does for

  callback tokens (ADR-0063);

- (d) store it in plaintext.

## Decision

Option (c), next to the file reference rather than instead of it.

1. **API.** `POST` and `PATCH /internal/partners[/{partnerOpId}]` accept `our_client_secret`

   (write-only, 1 to 4096 characters). Responses never contain it; they carry

   `our_client_secret_set: true` when an encrypted secret is stored. A `PATCH` without the field

   leaves the stored secret alone, a new value replaces it, and `null` removes it.

2. **One source per partner.** A partner has either `our_client_secret` or `our_client_secret_ref`.

   Sending both in one request is a `400`. Setting one on an existing partner drops the other,

   so an operator can move a partner from a mounted file to a typed secret and back. The rule

   that `our_client_id`, a secret and `token_endpoint` come as a set (`422` otherwise) counts

   either source as the secret.

3. **Storage.** A `partner_op_secrets` table, one row per partner (PK = FK `partner_op_id`,

   `ON DELETE CASCADE`), column `encrypted_secret` (BYTEA). It is separate from `partner_ops`

   so the ciphertext is read only when FM requests a token, not on every partner lookup. The

   partner row and its secret row are written in one transaction.

4. **Encryption.** AES-256-GCM, a fresh random 12-byte nonce per value, stored as

   `nonce ‖ ciphertext‖tag`, no associated data. The key is `FM_PARTNER_SECRET_KEY`: base64url of

   exactly 32 bytes, validated at startup. A rejected key is never echoed in the startup error.

   Rotation is not supported: a new key makes every stored secret unreadable, and token requests

   for those partners fail as a configuration error until the operator types the secrets again.

5. **No key configured → 501.** FM starts and logs a warning. A request that carries a

   non-null `our_client_secret` is rejected with `501` ProblemDetails

   (`partner-secret-storage-unavailable`) before anything is written. Partners using

   `our_client_secret_ref` are unaffected.

6. **Token acquisition.** The token provider reads the file when `our_client_secret_ref` is set,

   otherwise decrypts the stored secret. A replaced secret is picked up at the next token

   refresh, as a rewritten file already is.

7. **Never returned or logged.** The secret is excluded from responses, from validation error

   bodies and from the `repr` of every object that carries it. OEG relays the registry routes

   without logging bodies and passes FM's `501` through.

## Consequences

The Portal can register a working `active` partner end to end. A database dump alone doesn't

reveal the secrets; the dump plus `FM_PARTNER_SECRET_KEY` does. Every deployment that wants typed

secrets has to supply and keep a stable key (a Helm value backed by a Kubernetes Secret).

This amends:

- FM #21's done-criterion: secrets may now be accepted and stored, encrypted. "Never returned or

  logged" still holds.

- `fm/overview-and-architecture.md`, outbound credentials: the secret is in a mounted file **or**

  encrypted in `fm_db`.

- `fm/implementation-and-delivery.md`: the outbound adapter description, the "OAuth

  client-secret security" point (`fm_db` no longer stores only the reference), and the

  secret-handling test line ("never appear in PostgreSQL" becomes "never appear in PostgreSQL

  unencrypted").

- `fm/requirements-and-design.md`, `partner_ops` table: `our_client_secret_ref` is one of two

  secret sources, plus the new `partner_op_secrets` table.

Revisit triggers: a platform secret store, a key-rotation requirement, or the move to

`private_key_jwt` that the FM docs already plan, which removes per-partner secrets altogether.

* [ADR-0064](0064-portal-is-one-container-with-a-thin-backend.md) - the portal is one image: a FastAPI backend serves the built React app at `/` and its API at `/api/*`, and forwards calls without merging data; no Next.js server

* [ADR-0065](0065-portal-sessions-keyed-by-the-cookie.md) - portal login sessions live in PostgreSQL, tokens encrypted with a per-session key only the cookie holds; login attempts are single-use rows; a session lasts as long as Keycloak accepts its refresh token

* [ADR-0066](0066-api-clients-through-keycloak-client-registration.md) - developers' API clients are Keycloak clients registered by a narrow provisioner, products fixed at creation within the creator's grants, secret shown once, registration token encrypted under one app-wide key; a refused token on delete is reported; no reconciliation with Keycloak in Release 2

* [ADR-0067](0067-partner-client-secret-encrypted-in-fm-db.md) - FM accepts the partner client secret typed in through the registry and stores it AES-GCM encrypted in fm_db (`partner_op_secrets`) under one app-wide key, never returned; `our_client_secret_ref` files stay as the other source, one per partner; no key → 501 (amends FM #21's no-secret-accepted criterion)

**Inbound (middleware):** FastAPI middleware reads the `Authorization: Bearer` header, validates the JWT signature against Keycloak's JWKS endpoint (cached, refreshed on `kid` miss), and checks issuer, expiry, and the `fed-mgmt` scope. The validated token's client identity resolves to a partner via `partner_ops`.

**Outbound:** Callers depend only on `PartnerTokenProviderPort`. The current adapter reads the partner secret from the mounted file named by `our_client_secret_ref`, requests a token from `token_endpoint`, caches it until near expiry, and the EWBI client attaches it as a Bearer header:

**Outbound:** Callers depend only on `PartnerTokenProviderPort`. The current adapter reads the partner secret from the mounted file named by `our_client_secret_ref` or, when the operator typed the secret in through the registry, decrypts it from `partner_op_secrets` (ADR-0067); it then requests a token from `token_endpoint`, caches it until near expiry, and the EWBI client attaches it as a Bearer header:

```python

class PartnerTokenProviderPort(Protocol):

4. **PII minimization.** FM logs partner identifiers and API types but does not log subscriber-level PII (MSISDN, IP) except in the routing lookup path where it is necessary.

5. **OAuth client-secret security.** `fm_db` stores only `our_client_secret_ref`. The secret is read from a restricted mounted file on token refresh; Kubernetes Secrets or Vault-backed mounts provide production material and rotation without FM redeployment.

5. **OAuth client-secret security.** A partner's secret is either a restricted mounted file named by `our_client_secret_ref` (Kubernetes Secrets or Vault-backed mounts, rotated without FM redeployment) or a value typed in through the partner registry. The typed value is stored AES-256-GCM encrypted in `partner_op_secrets` under `FM_PARTNER_SECRET_KEY` and is never returned or logged; without the key FM answers 501 to a request carrying one (ADR-0067). Either way the secret is read on token refresh only.

### Q.3 Rate Limiting and Abuse Prevention

- **FM → SRM flow:** Use Docker Compose with FM, SRM (MockAdapter), PostgreSQL, and NATS. Verify that an inbound EWBI deploy request flows through to `event.srm.operation.completed`.

- **Inbound OAuth2:** Verify JWT signature, issuer, expiry, `fed-mgmt` scope, and `azp` → `oauth2_client_id` partner resolution against a test Keycloak realm.

- **Outbound OAuth2:** Verify token acquisition, near-expiry caching, concurrent refresh serialization, mounted-secret rotation, and Bearer-header injection against a mock partner Authorization Server.

- **Outbound OAuth2:** Verify token acquisition, near-expiry caching, concurrent refresh serialization, mounted-secret rotation, token acquisition with a stored encrypted secret, and Bearer-header injection against a mock partner Authorization Server.

- **DataBus integration:** Verify `command.srm.*` publishing and `event.srm.operation.completed` consumption round-trip.

### S.3 Contract Tests

- **Topology leakage detection:** Automated tests that send requests through the full FM pipeline and assert that no internal IP addresses, Kubernetes DNS names, cluster names, namespace names, or internal trace IDs appear in any EWBI response body or header.

- **Header stripping:** Verify that `X-Internal-*` headers from inbound EWBI requests are removed before processing.

- **Bearer-token attacks:** Verify forged signatures, `alg=none`, wrong issuer, expired tokens, absent scope, and unknown `azp` are rejected without leaking token or partner details.

- **Secret handling:** Verify client secrets never appear in PostgreSQL, exceptions, logs, or partner-facing errors and are reread after mounted-file rotation.

- **Secret handling:** Verify client secrets never appear unencrypted in PostgreSQL, nor in API responses, validation errors, exceptions, logs, or partner-facing errors, and are reread after mounted-file rotation or replacement through the registry.

### S.5 Mock Partner OP

|---|---|---|---|

| **Global routing table maintenance.** Maintaining an accurate, real-time map of MSISDN prefixes and IP ranges to global telco operators is extremely difficult. | High | High | Do not build this from scratch. Start with static operator-managed routing rules (< 50 partners). Integrate with GSMA Telco Finder service or aggregator routing table when available. |

| **Topology leakage.** A code change inadvertently exposes internal infrastructure details to a partner operator. | Medium | Critical | Automated topology leakage tests in CI. `TopologyHider` middleware runs on all EWBI responses. Security review gate for FM PRs touching response handling. |

| **Pairwise outbound client-secret management.** Client-secret authentication requires one secret per partner relationship and coordinated rotation. | Medium | High | Use mounted secret files now so onboarding/rotation needs no FM redeploy; move the `PartnerTokenProviderPort` implementation to `private_key_jwt` where partner Authorization Servers support it. |

| **Pairwise outbound client-secret management.** Client-secret authentication requires one secret per partner relationship and coordinated rotation. | Medium | High | Use mounted secret files, or secrets typed in through the registry and stored encrypted (ADR-0067), so onboarding/rotation needs no FM redeploy; move the `PartnerTokenProviderPort` implementation to `private_key_jwt` where partner Authorization Servers support it. |

| **Future mTLS certificate management at scale.** Sender-binding would add certificate issuance, revocation, and rotation across all partners. | Medium | Medium | Do not add certificate fields piecemeal. Define the RFC 8705 trust and lifecycle model before implementation. |

### T.2 Tradeoffs

2. **Scaffold the FM service:** Create the FastAPI project with hexagonal structure, integrate oop-commons (logging, health, DB, events), and configure Docker Compose entry.

3. **Implement `fm_db` schema:** Create Alembic migrations for `partner_ops` (OAuth2 client identities, token endpoint, mounted-secret reference), `federation_agreements`, `routing_rules`, and `federation_transactions` (`federation_operation_id` with partial unique index; nullable `operation_id` and `correlation_id` for sync-query audits).

3. **Implement `fm_db` schema:** Create Alembic migrations for `partner_ops` (OAuth2 client identities, token endpoint, mounted-secret reference), `partner_op_secrets` (encrypted typed secrets, ADR-0067), `federation_agreements`, `routing_rules`, and `federation_transactions` (`federation_operation_id` with partial unique index; nullable `operation_id` and `correlation_id` for sync-query audits).

4. **Build the inbound deploy flow:** Implement canonical OPG.04 `InstallApp` → Bearer-JWT validation → agreement check → capacity-check → persist transaction → `command.srm.service.deploy` publish → `event.srm.operation.completed` consumption → topology-stripped OPG.04 callback/resource response.

- `oauth2_client_id` identifies the partner at **our** Authorization Server and maps the inbound

  token's `azp` claim to one `partner_ops` row.

- `our_client_id` identifies our FM at the **partner's** Authorization Server. Its secret remains in

  a mounted file referenced by `our_client_secret_ref`, never inline in `fm_db`.

- `our_client_id` identifies our FM at the **partner's** Authorization Server. Its secret comes from

  one of two places (ADR-0067): a mounted file referenced by `our_client_secret_ref`, or a value the

  operator typed in through the partner registry, stored AES-256-GCM encrypted in

  `partner_op_secrets` and never returned.

**Inbound: partner OP calls our FM**

sequenceDiagram

    participant O as OEG / outbound service

    participant F as Our FM (OAuth2 client)

    participant S as Mounted partner secret

    participant S as Partner secret (mounted file or encrypted in fm_db)

    participant A as Partner Authorization Server

    participant P as Partner FM (Resource Server)

    O->>F: Forward request for partner

    Note over F: Reuse cached token when not near expiry

    F->>S: Read secret on token-cache miss

    F->>S: Read (and decrypt) secret on token-cache miss

    S-->>F: Secret for our_client_id

    F->>A: Client-Credentials request<br/>our_client_id + secret + fed-mgmt

    A-->>F: Signed access-token JWT

### Partner OP Profile

The external operator. Contains EWBI and OAuth2 endpoints, the partner's inbound client identity, our outbound client registration and mounted-secret reference, supported API families, SLA limits, and connection status. Each partner has exactly one current profile row in `partner_ops`; updates mutate that row in place and are audited via `updated_at` plus `federation_transactions` / operational logs (full immutable version history is out of scope for Release 2.0).

The external operator. Contains EWBI and OAuth2 endpoints, the partner's inbound client identity, our outbound client registration and its secret (a mounted-file reference, or an encrypted value in `partner_op_secrets`), supported API families, SLA limits, and connection status. Each partner has exactly one current profile row in `partner_ops`; updates mutate that row in place and are audited via `updated_at` plus `federation_transactions` / operational logs (full immutable version history is out of scope for Release 2.0).

**Maps to:** `partner_ops` table in `fm_db`.

| `base_url` | TEXT | NOT NULL | Partner's EWBI base URL |

| `oauth2_client_id` | VARCHAR(255) | NOT NULL, UNIQUE | Partner's client identity at our Authorization Server; matched against inbound JWT `azp` |

| `our_client_id` | VARCHAR(255) | | Our client identity at this partner's Authorization Server |

| `our_client_secret_ref` | TEXT | | Mounted-file reference for the secret backing `our_client_id`; secret is never stored inline |

| `our_client_secret_ref` | TEXT | | Mounted-file reference for the secret backing `our_client_id`. Mutually exclusive with a row in `partner_op_secrets` (ADR-0067) |

| `token_endpoint` | TEXT | | Partner Authorization Server token endpoint used for outbound Client-Credentials |

| `partner_ca_cert_path` | TEXT | | Optional CA bundle for ordinary HTTPS server-certificate verification; not an mTLS identity field |

| `supported_apis` | JSONB | NOT NULL, DEFAULT '[]' | Array of API family identifiers the partner supports |

- `idx_partner_ops_status` on `status`

- `idx_partner_ops_oauth2_client_id` on `oauth2_client_id`

#### `partner_op_secrets`

The secret backing `our_client_id` when the operator typed it in through the partner registry instead of mounting a file (ADR-0067). Kept apart from `partner_ops` so only the token provider reads the ciphertext.

| Column | Type | Constraints | Description |

|---|---|---|---|

| `partner_op_id` | UUID | PK, FK → partner_ops(id) ON DELETE CASCADE | Partner the secret belongs to |

| `encrypted_secret` | BYTEA | NOT NULL | AES-256-GCM under `FM_PARTNER_SECRET_KEY`: 12-byte nonce, then ciphertext and tag |

| `created_at` | TIMESTAMPTZ | NOT NULL, DEFAULT now() | |

| `updated_at` | TIMESTAMPTZ | NOT NULL, DEFAULT now() | |

#### `federation_contexts`

| Column | Type | Constraints | Description |