Merge branch 'docs/datalab-helm-deployment' into 'feat/datalab' (eec72fab) · Commits · OOP / architecture

data-lab/architecture.md

+63 −52

File changed.

Preview size limit exceeded, changes collapsed.

decisions/0069-data-lab-helm-chart-alternative-to-compose.md

0 → 100644

+64 −0

Changes for decisions/0069-data-lab-helm-chart-alternative-to-compose.md: 64 added lines, 0 removed lines.
Original line number	Diff line number	Diff line
		---
		type: Decision Record
		title: "ADR-0069: The Data Lab can also be deployed with the OOP Helm chart, as a minimal Kubernetes profile"
		description: "Subchart datalab of the OOP Helm chart deploys the Data Lab on Kubernetes, off by default, with the UI on a NodePort, one Virtuoso and the hub and search APIs cluster-internal."
		tags: [adr, data-lab, platform, deployment]
		timestamp: 2026-10-15T00:00:00Z
		status: "proposed"
		---

		# ADR-0069: The Data Lab can also be deployed with the OOP Helm chart, as a minimal Kubernetes profile

		Status: proposed
		Date: 2026-10-15
		Affects: [data-lab/architecture.md](/data-lab/architecture.md) §H.4; OOP Helm chart (`oop-platform-chart/charts/datalab`, realm `dataspace` in `charts/keycloak`)

		## Context

		ADR-0049 deploys the Data Lab as compose profile `data-lab` behind the platform Nginx (`/data-lab/*`) and
		Keycloak. OOP is also deployed with a Helm chart, and that deployment has no Nginx and no ingress: every
		component (OEG, SRM, portal, Keycloak) is reached on its own NodePort. The Data Lab needs a Helm deployment for the
		platforms that run OOP on Kubernetes.

		## Decision

		Subchart `datalab` of the OOP Helm chart, off by default (`datalab.enabled` together with
		`keycloak.realms.dataspace.enabled`):

		- Workloads: piveau hub-repo and hub-search, one Virtuoso (the hub's metrics and history graphs go to the main
		store; no shadow store), single-node Elasticsearch, the Data Lab UI and API, and a Job that creates the two
		catalogues.
		- Exposure: the UI only, on a NodePort (default 30083), at `/`; it forwards `/api/` to the Data Lab API. Hub,
		search, Virtuoso and Elasticsearch are cluster-internal; there is no SPARQL route.
		- Identity: realm `dataspace` on the platform Keycloak, provisioned by the OOP Keycloak chart from the same client
		definitions as the realm export in the Data Lab repo; client secrets are generated by the chart.
		- Images: the Data Lab UI and API images published by the Data Lab CI; the API reaches Keycloak at its in-cluster
		address (`KEYCLOAK_INTERNAL_URL`) while tokens are still checked against the public issuer.

		REQ-DL-13 (`/data-lab/*` through the platform Nginx) applies where there is a platform Nginx, that is the compose
		profile; in the Helm deployment the Data Lab is exposed like the other OOP components. The single Virtuoso differs from
		§H.1 for the Helm deployment only. The compose profile of ADR-0049 is unchanged.

		## Consequences

		What becomes easier:

		- The Data Lab deploys with the rest of OOP on Kubernetes, with one command and the same login.
		- About 3 GB of memory instead of 4 GB or more, and fewer moving parts.

		What becomes harder:

		- Machine clients (REQ-DL-11 API keys, contributor scripts) cannot reach the hub or search API from outside the
		cluster; they use `kubectl port-forward` until the hub is exposed.
		- The realm definition exists twice (Data Lab repo export, Keycloak chart template) and must be kept in step.
		- Records are minted with the Helm deployment's public address in `datalab.baseUri`; it must be set explicitly and
		never changed on a real deployment (MAP §12).

		Deferred or excluded:

		- `/data-lab/*` routing, external hub and search APIs, the shadow Virtuoso, NetworkPolicies for the stores.

		Revisit trigger:

		- A platform-wide ingress lands in the Helm chart, machine clients need the hub API from outside the cluster, or the
		metrics and history graphs outgrow the main store.

decisions/index.md

+1 −0

Changes for decisions/index.md: 1 added line, 0 removed lines.
Original line number	Diff line number	Diff line
		@@ -52,3 +52,4 @@ the reasoning lives here. Never renumber; supersede instead (see the record-deci
		* [ADR-0048](0048-data-lab-piveau-hub-and-6gdali-map.md) - Data Lab adopts piveau-hub as catalogue engine and the 6G-DALI Metadata Application Profile (DCAT-AP 3.0 + DQV + PROV-O) as normative metadata profile
		* [ADR-0049](0049-data-lab-standalone-profile-shared-ingress-and-iam.md) - Data Lab is a standalone compose profile behind the platform Nginx and Keycloak (realm `dataspace`); no DataBus/SRM/OEG coupling in the catalogue release
		* [ADR-0050](0050-data-lab-draft-version-not-pinned.md) - data-lab/ specs carry version 0.1 (Draft) and are excluded from the Release 2 pin (extends ADR-0021 exceptions)
		* [ADR-0069](0069-data-lab-helm-chart-alternative-to-compose.md) - The Data Lab can also be deployed with the OOP Helm chart: off by default, UI on a NodePort, one Virtuoso, hub and search cluster-internal (proposed)

platform/architecture.md

+3 −2

Changes for platform/architecture.md: 3 added lines, 2 removed lines.
Original line number	Diff line number	Diff line
		@@ -796,7 +796,7 @@ this stream or a dedicated stream is carved out is left to FM/platform-infra to
		\| TF-SDK → NEF \| 3GPP OAuth2 or mTLS \| TF-SDK adapter config \|
		\| TF-SDK → Kubernetes \| ServiceAccount token or kubeconfig \| TF-SDK adapter config \|
		\| Developer → Portal \| OIDC Authorization Code Flow with PKCE (Keycloak) \| Portal \|
		\| Contributor / consumer → Nginx → Data Lab \| OIDC Authorization Code + PKCE (Keycloak realm `dataspace`) for the UI; `X-API-Key` or client-credentials token for scripted registration; anonymous read (ADR-0049) \| Data Lab \|
		\| Contributor / consumer → Nginx → Data Lab \| OIDC Authorization Code + PKCE (Keycloak realm `dataspace`) for the UI; client-credentials token of the script's own Keycloak client for scripted registration (the hub API key stays inside the Data Lab API); anonymous read (ADR-0049) \| Data Lab \|

		### Roles

		@@ -843,7 +843,7 @@ All components use `oop-commons` structured JSON logging. Every log entry includ
		\| SRM \| Process alive \| PostgreSQL + DataBus connectivity \|
		\| FM \| Process alive \| PostgreSQL + DataBus connectivity \|
		\| Portal BFF \| Process alive \| OEG reachable + IAM reachable \|
		\| Data Lab (`piveau-hub-repo`) \| HTTP 200 on `/` \| Virtuoso + `piveau-hub-search` (Elasticsearch) reachable \|
		\| Data Lab (`piveau-hub-repo`) \| HTTP 200 on `/` \| Virtuoso + `piveau-hub-search` (Elasticsearch) reachable (planned; today HTTP 200 on `/`) \|

		### Key Metrics (Phase 2)

		@@ -1513,3 +1513,4 @@ Terminology is owned by the single platform glossary — see
		\| 2.0 \| 2026-09-01 \| Ownership Matrix corrected: RFC 7807 problem+json construction for SBI refusals is SRM's, not TF-SDK's. `can_serve_target` interface listing updated to the tri-state answer (ADR-0040) and TF-SDK-owned adapter factory (ADR-0041). \|
		\| 2.0 \| 2026-09-14 \| Data Lab (draft component, next release) registered: component overview row, boundary-matrix rows for the dataset catalogue and the reserved data-transfer responsibility, security and health-check rows, compose profile `data-lab`, scaling row, Phase 4 deliverables 7–8, Intelligence-domain cross-reference (ADR-0048, ADR-0049, ADR-0050). \|
		\| 2.0 \| 2026-09-04 \| Component Boundary Matrix updated: operator-triggered federation establishment is proxied through OEG (`command.fm.*`, ADR-0043); Edge Cloud Zone locality determination for deployment targets moves from OEG to SRM as an interim layer pending a future runtime-coordinator component (ADR-0044). Placement-decisions row absorbs the new locality row (was listing an inconsistent Not-Owner set); federation admin CRUD folded into the existing EWBI row instead of a redundant new one; Not-Owner columns corrected to name the passive consumer (FM) rather than an unrelated component. \|
		\| 2.0 \| 2026-10-16 \| Data Lab security row: scripts register with a client-credentials token of their own Keycloak client; the hub API key stays inside the Data Lab API. Data Lab readiness row marked planned (data-lab architecture REQ-DL-15). \|

platform/glossary.md

+2 −1

Changes for platform/glossary.md: 2 added lines, 1 removed line.
Original line number	Diff line number	Diff line
		@@ -157,6 +157,7 @@ This is the single glossary for all Open Operator Platform (OOP) architecture do
		\| 2.0 \| 2026-07-29 \| RFC 7807 entry corrected: no longer claims all OOP components use it. It is SRM's internal-API format only; OEG's northbound API uses CAMARA's `ErrorInfo` (ADR-0030). \|
		\| 2.0 \| 2026-08-05 \| Capability instance entry corrected: `external_ref` is the backend realization ref, not a caller-facing id — caller-facing identity lives on `service_instance.id` for every family (ADR-0033). \|
		\| 2.0 \| 2026-09-01 \| Placement, Network Domain, and Control path binding entries note the device-targeted control-path-probe exception (no zone/domain pin; adapter probing, ADR-0038/ADR-0040) and TF-SDK's adapter factory (ADR-0041). \|
		\| 2.0 \| 2026-10-16 \| Staging catalogue entry corrected: records are promoted to the production catalogue `oop-org`, not to `6g-dali*` catalogues. \|

		## Data Lab Concepts

		@@ -169,5 +170,5 @@ This is the single glossary for all Open Operator Platform (OOP) architecture do
		\| DQV \| W3C Data Quality Vocabulary; `dqv:QualityMeasurement` records per distribution carry pass/fail results of data-quality expectations. \|
		\| PROV-O \| W3C provenance ontology; `prov:wasDerivedFrom` links derived datasets to their sources. \|
		\| SHACL \| W3C Shapes Constraint Language; the MAP's layered `dali-*` shapes validate records before registration. \|
		\| Staging catalogue \| The Data Lab catalogue holding submitted, not-yet-reviewed records; promotion to the community (`6g-dali*`) catalogues is a steward action. \|
		\| Staging catalogue \| The Data Lab catalogue holding submitted, not-yet-reviewed records; promotion to the production catalogue (`oop-org`) is a steward action. \|
		\| Dataspace Protocol (DSP) / EDC \| IDSA protocol for catalogue exchange, contract negotiation and transfer, and its Eclipse reference connector — the future Data Lab connector (not in the catalogue release). \|

Original line number	Diff line number	Diff line
		@@ -796,7 +796,7 @@ this stream or a dedicated stream is carved out is left to FM/platform-infra to
		\| TF-SDK → NEF \| 3GPP OAuth2 or mTLS \| TF-SDK adapter config \|
		\| TF-SDK → Kubernetes \| ServiceAccount token or kubeconfig \| TF-SDK adapter config \|
		\| Developer → Portal \| OIDC Authorization Code Flow with PKCE (Keycloak) \| Portal \|
		\| Contributor / consumer → Nginx → Data Lab \| OIDC Authorization Code + PKCE (Keycloak realm `dataspace`) for the UI; `X-API-Key` or client-credentials token for scripted registration; anonymous read (ADR-0049) \| Data Lab \|
		\| Contributor / consumer → Nginx → Data Lab \| OIDC Authorization Code + PKCE (Keycloak realm `dataspace`) for the UI; client-credentials token of the script's own Keycloak client for scripted registration (the hub API key stays inside the Data Lab API); anonymous read (ADR-0049) \| Data Lab \|

		### Roles

		@@ -843,7 +843,7 @@ All components use `oop-commons` structured JSON logging. Every log entry includ
		\| SRM \| Process alive \| PostgreSQL + DataBus connectivity \|
		\| FM \| Process alive \| PostgreSQL + DataBus connectivity \|
		\| Portal BFF \| Process alive \| OEG reachable + IAM reachable \|
		\| Data Lab (`piveau-hub-repo`) \| HTTP 200 on `/` \| Virtuoso + `piveau-hub-search` (Elasticsearch) reachable \|
		\| Data Lab (`piveau-hub-repo`) \| HTTP 200 on `/` \| Virtuoso + `piveau-hub-search` (Elasticsearch) reachable (planned; today HTTP 200 on `/`) \|

		### Key Metrics (Phase 2)

		@@ -1513,3 +1513,4 @@ Terminology is owned by the single platform glossary — see
		\| 2.0 \| 2026-09-01 \| Ownership Matrix corrected: RFC 7807 problem+json construction for SBI refusals is SRM's, not TF-SDK's. `can_serve_target` interface listing updated to the tri-state answer (ADR-0040) and TF-SDK-owned adapter factory (ADR-0041). \|
		\| 2.0 \| 2026-09-14 \| Data Lab (draft component, next release) registered: component overview row, boundary-matrix rows for the dataset catalogue and the reserved data-transfer responsibility, security and health-check rows, compose profile `data-lab`, scaling row, Phase 4 deliverables 7–8, Intelligence-domain cross-reference (ADR-0048, ADR-0049, ADR-0050). \|
		\| 2.0 \| 2026-09-04 \| Component Boundary Matrix updated: operator-triggered federation establishment is proxied through OEG (`command.fm.*`, ADR-0043); Edge Cloud Zone locality determination for deployment targets moves from OEG to SRM as an interim layer pending a future runtime-coordinator component (ADR-0044). Placement-decisions row absorbs the new locality row (was listing an inconsistent Not-Owner set); federation admin CRUD folded into the existing EWBI row instead of a redundant new one; Not-Owner columns corrected to name the passive consumer (FM) rather than an unrelated component. \|
		\| 2.0 \| 2026-10-16 \| Data Lab security row: scripts register with a client-credentials token of their own Keycloak client; the hub API key stays inside the Data Lab API. Data Lab readiness row marked planned (data-lab architecture REQ-DL-15). \|

Original line number	Diff line number	Diff line
		@@ -157,6 +157,7 @@ This is the single glossary for all Open Operator Platform (OOP) architecture do
		\| 2.0 \| 2026-07-29 \| RFC 7807 entry corrected: no longer claims all OOP components use it. It is SRM's internal-API format only; OEG's northbound API uses CAMARA's `ErrorInfo` (ADR-0030). \|
		\| 2.0 \| 2026-08-05 \| Capability instance entry corrected: `external_ref` is the backend realization ref, not a caller-facing id — caller-facing identity lives on `service_instance.id` for every family (ADR-0033). \|
		\| 2.0 \| 2026-09-01 \| Placement, Network Domain, and Control path binding entries note the device-targeted control-path-probe exception (no zone/domain pin; adapter probing, ADR-0038/ADR-0040) and TF-SDK's adapter factory (ADR-0041). \|
		\| 2.0 \| 2026-10-16 \| Staging catalogue entry corrected: records are promoted to the production catalogue `oop-org`, not to `6g-dali*` catalogues. \|

		## Data Lab Concepts

		@@ -169,5 +170,5 @@ This is the single glossary for all Open Operator Platform (OOP) architecture do
		\| DQV \| W3C Data Quality Vocabulary; `dqv:QualityMeasurement` records per distribution carry pass/fail results of data-quality expectations. \|
		\| PROV-O \| W3C provenance ontology; `prov:wasDerivedFrom` links derived datasets to their sources. \|
		\| SHACL \| W3C Shapes Constraint Language; the MAP's layered `dali-*` shapes validate records before registration. \|
		\| Staging catalogue \| The Data Lab catalogue holding submitted, not-yet-reviewed records; promotion to the community (`6g-dali*`) catalogues is a steward action. \|
		\| Staging catalogue \| The Data Lab catalogue holding submitted, not-yet-reviewed records; promotion to the production catalogue (`oop-org`) is a steward action. \|
		\| Dataspace Protocol (DSP) / EDC \| IDSA protocol for catalogue exchange, contract negotiation and transfer, and its Eclipse reference connector — the future Data Lab connector (not in the catalogue release). \|