Commit fb1ad706 authored by amaxilat's avatar amaxilat
Browse files

docs(data-lab): add Data Lab architecture spec and supporting ADRs 

Introduces the Data Lab component (piveau-hub based dataspace metadata
catalogue, DCAT-AP 3.0 / 6G-DALI MAP) as the next OOP release's data- and
AI-asset pillar: architecture.md, component index, ADR-0048..0050, and the
vendored MAP spec under external-specs/. Wires the component into the
top-level README/index/decisions/platform docs.

Co-Authored-By: default avatarClaude Sonnet 5 <noreply@anthropic.com>
parent c89c2e37

README.md

+2 −1
Changes for README.md: 2 added lines, 1 removed line.
Original line number Diff line number Diff line
@@ -22,7 +22,7 @@ Release 2 specs are pinned at `version: "2.0"` in frontmatter until the next rel 
**Implementing a component**



1. Read [platform/architecture.md](platform/architecture.md) for the end-to-end mental model.

2. Open the component folder ([oeg/](oeg/), [srm/](srm/), [fm/](fm/), [tf-sdk/](tf-sdk/)) via its `index.md`.

2. Open the component folder ([oeg/](oeg/), [srm/](srm/), [fm/](fm/), [tf-sdk/](tf-sdk/), [data-lab/](data-lab/)) via its `index.md`.

3. Follow architecture → interface contract → persistence model → flow walk-throughs in that order.

4. When a spec cites an ADR, read the decision record for the rationale behind the rule.
@@ -47,6 +47,7 @@ Release 2 specs are pinned at `version: "2.0"` in frontmatter until the next rel 
| [srm/](srm/) | Service Resource Manager — orchestration and inventory |

| [fm/](fm/) | Federation Manager — cross-operator EWBI federation |

| [tf-sdk/](tf-sdk/) | Transformation Function SDK — vendor adapters inside SRM |

| [data-lab/](data-lab/) | Data Lab — dataspace metadata catalogue (draft, ADR-0050) |

| [decisions/](decisions/) | Architecture Decision Records |

| [external-specs/](external-specs/) | Vendored third-party API definitions |

| [archive/](archive/) | Superseded working documents |

data-lab/architecture.md

0 → 100644
+432 −0

File added.

Preview size limit exceeded, changes collapsed.

data-lab/index.md

0 → 100644
+12 −0
Changes for data-lab/index.md: 12 added lines, 0 removed lines.
Original line number Diff line number Diff line 
# Data Lab (DL)



Draft component (`version: "0.1 (Draft)"`, ADR-0050). Read the architecture first; the vendored

metadata profile is the normative description model.



# Normative metadata reference



* [6G-DALI Metadata Application Profile (vendored snapshot)](/external-specs/6gdali-metadata-application-profile/VERSION.md) - DCAT-AP 3.0 + `dali:` testbed context + PROV-O + DQV, with layered SHACL shapes and the `dali:` ontology (ADR-0026, ADR-0048)



# Core specs



* [Architecture](architecture.md) - the dataspace metadata catalogue (piveau-hub + dataspace-ui): mental model, use cases, scope, interfaces, deployment behind the platform Nginx/Keycloak, and the evolution path to connector, storage and pipelines

decisions/0048-data-lab-piveau-hub-and-6gdali-map.md

0 → 100644
+48 −0
Changes for decisions/0048-data-lab-piveau-hub-and-6gdali-map.md: 48 added lines, 0 removed lines.
Original line number Diff line number Diff line 
---

type: Decision Record

title: "ADR-0048: Data Lab adopts piveau-hub as catalogue engine and the 6G-DALI MAP as normative metadata profile"

description: "The OOP Data Lab catalogue reuses piveau-hub (repo + search) and the 6G-DALI Metadata Application Profile (DCAT-AP 3.0 + DQV + PROV-O) rather than a purpose-built OOP catalogue service."

tags: [adr, data-lab]

timestamp: 2026-09-14T00:00:00Z

status: "accepted"

---



# ADR-0048: Data Lab adopts piveau-hub as catalogue engine and the 6G-DALI MAP as normative metadata profile



**Status:** accepted

**Date:** 2026-09-14

**Affects:** [data-lab/architecture.md](/data-lab/architecture.md), [external-specs/6gdali-metadata-application-profile/VERSION.md](/external-specs/6gdali-metadata-application-profile/VERSION.md), [platform/architecture.md](/platform/architecture.md)



## Context



The next OOP release introduces the Data Lab, starting with a dataset catalogue. Options:

(a) build an OOP-native catalogue service on the `python-template` (FastAPI + PostgreSQL) with an

OOP-defined dataset schema; (b) reuse the piveau-hub based catalogue and the DCAT-AP 3.0

Metadata Application Profile already developed and operated in 6G-DALI; (c) reuse piveau but

define a new OOP metadata profile.



A native service would be simpler to fit into the OOP stack but would have to re-implement

DCAT-AP conformance, SHACL validation, SPARQL and search, and would not be federatable with

data.europa.eu / GAIA-X style catalogues. A new profile would fork the description model the

6G-DALI testbeds already publish against.



## Decision



Option (b). The Data Lab catalogue is **piveau-hub** (`piveau-hub-repo`, `piveau-hub-search`,

Virtuoso, Elasticsearch) plus the 6G-DALI `dataspace-ui`, deployed as pinned upstream images with

OOP configuration and branding. The **6G-DALI Metadata Application Profile** is the normative

description model for datasets, distributions and data quality; it is vendored under

`external-specs/` with commit pinning (ADR-0026) and is not restated in OOP specs. OOP-specific

needs are expressed as profile extensions, never as a competing schema.



The catalogue is **not** built on the `python-template` / `oop-commons` stack; the platform's

"Python everywhere" decision applies to OOP-authored services, not to vendored products.



## Consequences



Easier: standards conformance, SPARQL/search/validation out of the box, direct continuity with

the 6G-DALI deployment and its contributors, future federation. Harder: a Java/Vert.x product in

an otherwise Python platform (separate logging/metrics conventions, no `oop-commons`), profile

changes require a re-vendor cycle, and piveau's own auth model must be mapped onto the platform

Keycloak. Revisit if piveau stops tracking DCAT-AP releases or if OOP needs catalogue semantics

piveau cannot express.

decisions/0049-data-lab-standalone-profile-shared-ingress-and-iam.md

0 → 100644
+39 −0
Changes for decisions/0049-data-lab-standalone-profile-shared-ingress-and-iam.md: 39 added lines, 0 removed lines.
Original line number Diff line number Diff line 
---

type: Decision Record

title: "ADR-0049: Data Lab runs as a standalone compose profile behind the platform Nginx and Keycloak, with no DataBus/SRM/OEG coupling"

description: "The Data Lab catalogue is deployed as its own profile, routed under /data-lab/* by the platform Nginx and authenticated by a dedicated realm on the platform Keycloak; it publishes no DataBus subjects and calls no OOP internal APIs in the catalogue release."

tags: [adr, data-lab, platform]

timestamp: 2026-09-14T00:00:00Z

status: "accepted"

---



# ADR-0049: Data Lab runs as a standalone compose profile behind the platform Nginx and Keycloak, with no DataBus/SRM/OEG coupling



**Status:** accepted

**Date:** 2026-09-14

**Affects:** [data-lab/architecture.md](/data-lab/architecture.md), [platform/architecture.md](/platform/architecture.md) (compose profiles, boundary matrix, security table)



## Context



The Data Lab catalogue must ship in the next release without destabilising OEG/SRM/FM. Options:

(a) fully standalone — own Keycloak and own ingress, merely documented as an OOP module;

(b) standalone services sharing the platform's Nginx and Keycloak, no application-level coupling;

(c) integrated — OEG catalogue read-through, Portal BFF aggregation, DataBus events for

registrations.



## Decision



Option (b). The Data Lab is compose profile `data-lab`, depends on profile `core` only for

**Nginx** (route family `/data-lab/*`) and **Keycloak** (dedicated realm `dataspace`; roles

`dl-contributor`, `dl-steward`). It owns its persistence (Virtuoso, Elasticsearch) under the

per-component isolation rule (ADR-0025), publishes and consumes **no** DataBus subjects, exposes

**no** `/internal/*` surface, and calls **no** OEG/SRM/FM API. Portal integration is a link.



## Consequences



Easier: zero change to existing components; one IAM and one ingress for operators; the Data

Lab can be enabled or disabled per deployment. Harder: no platform-wide view of datasets from

the Portal yet; two catalogues (SRM service catalogue, Data Lab dataset catalogue) with no

cross-reference. Revisit when the connector and DataOps land — registrations and quality

results are natural `event.datalab.*` candidates, and a TF-SDK Intelligence-domain

`data_product_query` capability is the intended path for OOP services to consume the catalogue.