Merge branch 'docs/callback-credentials' into 'develop'

---

type: Decision Record

title: "ADR-0063: Client callback tokens are stored AES-GCM encrypted in oeg_db under one app-wide key; a token that is gone is never replaced by an unauthenticated delivery"

description: "EAM sinkCredential and QoD notificationAuthToken are kept in a callback_credentials table, encrypted with AES-256-GCM under a single configured key. Without the key OEG rejects tokens with 501. Expired or revoked tokens stay as tombstones so their deliveries are skipped and recorded, never sent without auth."

tags: [adr]

timestamp: 2026-10-06T00:00:00Z

status: "proposed"

---

# ADR-0063: Client callback tokens are stored encrypted in `oeg_db` under one app-wide key

**Status:** proposed

**Date:** 2026-10-06

**Affects:** amends [ADR-0006](/decisions/0006-oeg-db-schema-consolidation.md) (`callback_registrations.sink_credential_ref`), [oeg/persistence-model.md](/oeg/persistence-model.md), [oeg/app-instance-flow.md](/oeg/app-instance-flow.md), [oeg/app-deployment-flow.md](/oeg/app-deployment-flow.md), [oeg/interface-contract.md](/oeg/interface-contract.md), [oeg/architecture.md](/oeg/architecture.md)

## Context

Two CAMARA families let the client hand OEG a bearer token that OEG must present on every

notification it later sends to the client's sink:

| Family | Request field | Expiry supplied by the client |

|---|---|---|

| EAM (vendored 0.1.0-alpha.1) | `subscriptionRequest.sinkCredential` (`AccessTokenCredential`: `credentialType=ACCESSTOKEN`, `accessToken`, `accessTokenType=bearer`, `accessTokenExpiresUtc` required) | Yes |

| QoD (vendored v0.10.1) | `webhook.notificationAuthToken` | No |

Notifications are sent from the event consumer, minutes to hours after the request, with no

client request in flight. ADR-0006 gave `callback_registrations` a `sink_credential_ref` column

holding a "secret reference", but OOP has no secret store for that reference to point at, so the

column was never populated and has been dropped from the code. Options:

- (a) an external secret store (Vault, Kubernetes Secrets written at runtime): a new

  infrastructure dependency for one table's worth of data;

- (b) encrypt the token into `oeg_db` with one key from OEG configuration;

- (c) a per-request key, as the Portal's session crypto does: impossible, since there is no

  request to carry the key at delivery time;

- (d) store the token in plaintext.

## Decision

Option (b).

1. **Storage.** One `callback_credentials` row per `callback_registrations` row that carried a

   token (PK = FK `callback_registration_id`). It is a separate table rather than a column

   because OEG's schema bootstrap (`create_all`) never alters an existing table, and it keeps the

   secret out of the row every delivery path reads. Columns: `encrypted_token` (BYTEA,

   NULLABLE), `expires_at` (TIMESTAMPTZ, NULLABLE, indexed), `created_at`.

2. **Encryption.** AES-256-GCM. Each token is encrypted with a fresh random 12-byte nonce and

   stored as `nonce ‖ ciphertext‖tag`; no associated data. The key is

   `CALLBACK_SETTINGS__SECRET_KEY`: base64url of exactly 32 bytes, validated at startup. One key

   serves the whole application. Rotation is not supported: changing the key makes every stored

   token undecryptable, and those deliveries are skipped (rule 5).

3. **No key configured → 501.** If the key is unset, a request carrying `sinkCredential` or

   `notificationAuthToken` is rejected with `501 NOT_IMPLEMENTED` before anything is persisted

   or published. Requests without a token are unaffected. OEG logs a warning at startup. Accepting

   a token OEG cannot store would leave the client waiting for authenticated notifications that

   never come. Both vendored specs declare 501.

4. **Expiry.**

   - EAM: `expires_at = accessTokenExpiresUtc`. A value that is not in the future is rejected

     `400 INVALID_ARGUMENT`.

   - QoD v0.10.1: the token has no expiry, so it expires with the session:

     `expires_at = session start + duration + 1 h`. The hour of grace covers the session's final

     notifications (SRM-reported expiry, delete), which arrive after the nominal end.

5. **Delivery rule.** At each delivery OEG looks up the registration's credential:

   | Credential state | Delivery |

   |---|---|

   | No row (client never sent a token) | Sent without `Authorization` |

   | Row, token present, not expired | Sent with `Authorization: Bearer <token>` |

   | Row, `expires_at` ≤ now | **Skipped** — `credential_expired` |

   | Row, token wiped (rule 6) | **Skipped** — `credential_revoked` |

   | Row, token cannot be decrypted or no key configured | **Skipped** — `credential_unusable` |

   A skipped delivery writes a `callback_deliveries` row with `state=skipped` and the reason in

   `last_error`. OEG never downgrades a registration that had a token to an unauthenticated

   delivery. This is the behaviour the vendored EAM spec describes for `accessTokenExpiresUtc`:

   "If the token expires while the resource is still active, the client will stop receiving

   notifications."

6. **Revocation keeps a tombstone.** When OEG no longer needs a token it sets

   `encrypted_token = NULL` and keeps the row, so the rule-5 lookup can still tell "never had a

   token" from "had one that is gone":

   - the reconciliation pass (every `RECONCILIATION_SETTINGS__INTERVAL_SECONDS`, default 60 s)

     wipes every token whose `expires_at` has passed;

   - EAM: after the notification that finds every instance under the subscription

     `terminated`;

   - QoD: after the `DELETE_REQUESTED` notification on `DELETE /sessions/{id}`.

7. **The token never leaves OEG except to the sink.** EAM responses carry no `sinkCredential`;

   the QoD `201` returns `webhook.notificationUrl` without `notificationAuthToken`, and

   `GET /sessions/{id}` returns no `webhook`. The token fields are excluded from logs. The sink

   must be `https://` on both families: the EAM `sink` pattern is from the vendored spec, and

   QoD's `notificationUrl` gets the same `^https://.+$` check. That is stricter than v0.10.1's

   `format: uri`, and matches QoD v1 `sink` and Commonalities.

8. **Error codes follow the vendored specs.** A `credentialType` other than `ACCESSTOKEN`, or a

   malformed credential, is `400 INVALID_ARGUMENT`. Commonalities' `INVALID_CREDENTIAL` is not

   among the vendored EAM 0.1.0-alpha.1 codes (`INVALID_ARGUMENT`, `OUT_OF_RANGE`), so OEG does

   not emit it yet (ADR-0015).

## Consequences

No new infrastructure: tokens live in `oeg_db` beside the registration they belong to, and a

database dump alone does not reveal them; the dump **plus** the key does. Every deployment that

wants authenticated callbacks must supply and keep a stable 32-byte key. Losing or changing it

silently degrades every live authenticated subscription to skipped deliveries (visible as

`credential_unusable` rows). The client gets no in-band signal that notifications stopped. CAMARA's

`subscription-ended` / `ACCESS_TOKEN_EXPIRED` belongs to explicit subscription management, which

the vendored EAM version does not define (same trigger as ADR-0008).

Revisit triggers: a platform secret store or a key-rotation requirement (rules 1–2); moving QoD

to v1, whose `sinkCredential` carries `accessTokenExpiresUtc`, so the client's expiry replaces

the session + 1 h rule (rule 4); a vendored EAM version that lists `INVALID_CREDENTIAL` (rule 8);

CAMARA explicit subscriptions (rule 5's silent stop becomes a `subscription-ended` event).

* [ADR-0047](0047-terminal-record-read-follows-camara-schema.md) - terminal-record read behaviour follows each family's own vendored CAMARA schema: Traffic Influence returns 200 with `state: deleted`, QoD stays 404 (amends ADR-0034's blanket 404 rule and its 204 delete code)

* [ADR-0048](0048-platform-roles.md) - three realm roles (app-provider, operator, platform-admin); new users get app-provider but call only the API products granted to them (each CAMARA scope gated by a product role); tenancy is the `organization` claim, else `sub`; development mode switches login off on localhost (amends ADR-0006's tenant identity and ADR-0035's entitlement note)

* [ADR-0062](0062-zone-state-uses-camara-inactive.md) - `zone.state` uses CAMARA's `active`/`inactive`/`unknown` (was `offline`), so OEG passes the zone status filter and value through unmapped; domain, capability and control-path-binding states keep `offline`

* [ADR-0063](0063-callback-tokens-encrypted-in-oeg-db.md) - client callback tokens (EAM sinkCredential, QoD notificationAuthToken) are stored AES-GCM encrypted in oeg_db under one app-wide key; no key → 501; expired/revoked tokens are tombstoned and their deliveries skipped, never sent unauthenticated (amends ADR-0006's sink_credential_ref)

 No newline at end of file

         the deploy command's service_specification_id (ADR-0011) — appDeploymentName

         from the request body, edgeCloudZones[])

       - one app_instances row per zone (state=instantiating, app_deployment_id set)

 10. Store callback if subscriptionRequest.sink present

 10. Store callback if subscriptionRequest.sink present (callback_registrations, plus an

     encrypted callback_credentials row if sinkCredential is present — ADR-0063).

     A sinkCredential with no CALLBACK_SETTINGS__SECRET_KEY configured is rejected 501

     NOT_IMPLEMENTED before step 9, so nothing is written or published.

 11. Publish CommandEnvelopeV1 + SrmServiceDeployV1 with N-entry targets[] (DataBus)

        subject: command.srm.service.deploy

      │

| Version | Date | Summary |

|---------|------|---------|

| 2.0 | 2026-07-07 | OOP Release 2.0. |

| 2.0 | 2026-10-06 | Step 10 stores the `sinkCredential` encrypted in `callback_credentials` and rejects it `501` when no callback key is configured (ADR-0063). |

 No newline at end of file

  tenant_id           = <from JWT>

  api_family          = "edge-application-management"

  sink                = <from subscriptionRequest.sink>

  sink_credential_ref = <secret reference to stored sinkCredential — never the raw token>

  event_types         = <from subscriptionRequest.types>

  expires_at          = <from config.subscriptionExpireTime, if present>

  is_active           = true

callback_credentials                        ← only if subscriptionRequest.sinkCredential present (ADR-0063)

  callback_registration_id = <callback_registrations.id>

  encrypted_token          = <AES-GCM(sinkCredential.accessToken) under CALLBACK_SETTINGS__SECRET_KEY>

  expires_at               = <sinkCredential.accessTokenExpiresUtc>

```

> If `sinkCredential` is present and OEG has no `CALLBACK_SETTINGS__SECRET_KEY`, the request is

> rejected `501 NOT_IMPLEMENTED` **before** any row is written or the command published. A

> `credentialType` other than `ACCESSTOKEN`, or an `accessTokenExpiresUtc` that is not in the

> future, is `400 INVALID_ARGUMENT` (ADR-0063).

> `config.subscriptionMaxEvents` and `config.subscriptionDetail`/`initialEvent` are accepted in

> the request payload but not yet mapped to any stored column in v1 — they are accepted and

> ignored.

`status: failed` (the id was assigned at request time; the instance just never reached `ready`).

`operationId` is never included — it is not a CAMARA field (ADR-0005, ADR-0008).

**Authorization (ADR-0063):** with no `sinkCredential` on the request, the callback is sent

without an `Authorization` header. With one, OEG decrypts the stored token and sends it as

`Bearer`. If that token has expired or been revoked, or cannot be decrypted, the callback is

**skipped**, never sent unauthenticated, and a `callback_deliveries` row records

`state=skipped`. Once every instance under the subscription is `terminated`, OEG revokes the

stored token.

---

# AppInstanceStatus Mapping

| Version | Date | Summary |

|---------|------|---------|

| 2.0 | 2026-07-07 | OOP Release 2.0. |

| 2.0 | 2026-10-06 | §6.3: `sink_credential_ref` replaced by a `callback_credentials` row (encrypted token + expiry), plus the 501/400 rules for `sinkCredential`; Response 4 states when callbacks carry, omit or skip `Authorization` (ADR-0063). |

 No newline at end of file

| REQ-OEG-05 | For long-running operations, the system shall persist an operation record in `oeg_db` with status `PENDING` under a unique internal `operationId`, publish a `command.srm.*` message on the DataBus matching `CommandEnvelopeV1` + payload schemas shared with SRM, and return HTTP 202 with the CAMARA resource body (`AppInstanceInfo` or `{appDeploymentId}`); `operationId` is exposed only via `/platform/v1/operations/{operationId}` (ADR-0005). | OPG.02 v10.0, OPG.10 v2.0 |

| REQ-OEG-06 | The system shall consume `event.srm.operation.completed` events from the DataBus and shall update the matching `operations` row to `COMPLETED`, `PARTIALLY_COMPLETED`, or `FAILED`, including error details from the payload (ADR-0005). | OPG.02 v10.0 |

| REQ-OEG-07 | The system shall perform idempotent publishes: the same logical request (same `operationId` or client `Idempotency-Key`) shall not create duplicate `PENDING` rows or duplicate `command.srm.*` publishes. | CAMARA Commonalities r4.3 |

| REQ-OEG-08 | When a request carries `subscriptionRequest.sink`, the system shall deliver status-change notifications as CloudEvents (`application/cloudevents+json`) per the vendored CAMARA EAM spec on terminal operation state, authenticating to the sink with `Authorization: Bearer` using the registered `sinkCredential` (ADR-0008). `operationId` is never included in callback payloads. | OPG.02 v10.0, CAMARA Commonalities r4.3 |

| REQ-OEG-08 | When a request carries `subscriptionRequest.sink`, the system shall deliver status-change notifications as CloudEvents (`application/cloudevents+json`) per the vendored CAMARA EAM spec on terminal operation state, authenticating to the sink with `Authorization: Bearer` using the registered `sinkCredential` (ADR-0008), stored encrypted at rest; a delivery whose token has expired or been revoked is skipped, never sent unauthenticated (ADR-0063). `operationId` is never included in callback payloads. | OPG.02 v10.0, CAMARA Commonalities r4.3 |

| REQ-OEG-09 | **Deferred to a later release:** the system shall enforce rate limits per tenant and API product, from the quota in `api_subscriptions` (ADR-0048). Release 2 enforces no rate limits. | OPG.02 v10.0 |

| REQ-OEG-10 | The system shall emit structured JSON logs including `x_correlator`, `operation_id`, `tenant_id`, `app_provider_id`, `client_id`, HTTP method/path, and outcome. | Platform baseline |

| REQ-OEG-11 | The system shall record API usage for billing/analytics in `api_usage_records` with timestamp, API family, HTTP status, and bytes estimate. | OPG.02 v10.0 |

| 2.0 | 2026-09-04 | UC-OEG-05 split by target type: zone-based deployment locality moves to SRM (ADR-0044), subscriber-identifier routing unchanged. Added UC-OEG-06 and REQ-OEG-17 for the OEG-proxied federation-establishment trigger (ADR-0043). Zone-based flow's cross-references corrected from UC-SRM-03/§G3 (now inbound/subscriber-identifier-only) to UC-SRM-03a/§G4. |

| 2.0 | 2026-10-04 | Auth responsibility and REQ-OEG-02 add per-operation CAMARA scope checks; F4 resolved for tenancy (`organization` claim, else `sub`), device mapping still open; F5 records that tenant scoping of reads and deletes is deferred; REQ-OEG-09 rate limits deferred and keyed by tenant and API product (ADR-0048). |

| 2.0 | 2026-10-04 | Auth and logging match the OEG implementation: no `oop-commons`; the CAMARA scope is required for every token, including QoD sessions (stricter than QoD v0.10.1). REQ-OEG-10 and §Structured Logging list the fields OEG logs (caller `tenant_id`, `app_provider_id`, `client_id`; one `http_request` line per request) and what is not logged yet (`operation_id`). |

| 2.0 | 2026-10-06 | REQ-OEG-08: the sink token is stored encrypted, and expired/revoked tokens skip the delivery (ADR-0063). |