@@ -68,6 +71,7 @@ If you are new to the Data Lab, this is the simplest useful way to think about i
**Key facts to remember:**
-`piveau-hub-repo` is the write path and the source of truth (RDF in Virtuoso); `piveau-hub-search` is the read/search path (Elasticsearch index kept in sync by the repo).
- The UI talks only to the Data Lab API. The hub API key never leaves the Data Lab API; scripts call the hub with their own Keycloak token.
- The MAP (DCAT-AP 3.0 + `dali:` extensions + DQV + PROV-O) is the normative description model; SHACL shapes are how it is enforced.
- Data quality lives on the **distribution** (`dqv:hasQualityMeasurement`), not on the dataset.
- Nothing in the Data Lab touches `oeg_db`, `srm_db`, `fm_db` or the DataBus.
@@ -93,8 +97,8 @@ This table answers: "What does the current architecture require the Data Lab to
| --- | --- |
| **Catalogue engine** | piveau-hub (`piveau-hub-repo` + `piveau-hub-search`) as the DCAT-AP 3.0 catalogue: catalogues, datasets, distributions, catalogue records. RDF persisted in Virtuoso; search index in Elasticsearch (ADR-0048). |
| **Metadata profile** | The 6G-DALI MAP is the normative description model: DCAT-AP 3.0 core fields, `dali:` 5G/6G testbed context (CMT), PROV-O provenance, W3C DQV quality on distributions. The vendored copy under `external-specs/` is the contract; this document does not restate its field tables. |
| **Registration API** | `PUT /datasets/{dataset-id}?catalogue={catalogue-id}` and catalogue CRUD on `piveau-hub-repo` with `text/turtle` or `application/ld+json` bodies (Registration API). Machine callers authenticate with an API key; interactive callers with a Keycloak bearer token. |
| **Validation** | *Target:* incoming records validated against the MAP (SHACL) before persistence, non-conformant records rejected with the report. *Today:*none beyond RDF syntax — the piveau profile mode that would validate broke indexing in hub-search 5.3.1 (not re-tested on 5.3.11) and is off, as in the 6G-DALI production stack (open question F3). |
| **Registration API** | `PUT /datasets/{dataset-id}?catalogue={catalogue-id}` and catalogue CRUD on `piveau-hub-repo` with `text/turtle` or `application/ld+json` bodies (Registration API). Every caller presents a Keycloak bearer token: scripts a client-credentials token of their own Keycloak client, whose service account is a member of the catalogues it writes to. The hub API key is held by the Data Lab API only. |
| **Validation** | *Target:* incoming records validated against the MAP (SHACL) before persistence, non-conformant records rejected with the report. *Today:*records created or edited through the Data Lab API are validated by it (REQ-DL-22); records PUT straight to the Registration API get no check beyond RDF syntax — the piveau profile mode that would validate broke indexing in hub-search 5.3.1 (not re-tested on 5.3.11) and is off, as in the 6G-DALI production stack (open question F3). |
| **Discovery** | Keyword and faceted search over the index through the Data Lab API's anonymous `/public/*` endpoints (and `/browse/*` for signed-in users, which adds their own catalogues), and the dataset page showing distributions, access/download links, licence, provenance and per-distribution quality results. Private catalogues (staging among them) are never part of public discovery, and piveau itself refuses their datasets to anyone without rights, so the same holds for direct calls to the hub and the search service. There is no public SPARQL endpoint (§H.5). |
| **Catalogue structure** | Each OOP deployment (one per partner) is bootstrapped with two catalogues: `oop-org` (production, platform-branded title) and `oop-org-staging` for submitted-but-not-yet-promoted records. Further catalogues (for example one per testbed) are created on demand through the Data Lab API; there are no per-partner sub-catalogues and no cross-deployment federation. Promotion staging → production is a manual steward action in this release. |
| **Web UI** | The Data Lab UI (React 18 / TypeScript / Vite, SB Admin 2 theme), one app: public pages (faceted search over catalogue, theme, format, licence, keyword and publisher; catalogue pages; dataset pages with distributions, testbed context and quality results) and, after sign-in, the management console (§H.7). Hash-routed and relative-pathed, so it works under any URL prefix. |
@@ -134,11 +138,11 @@ Give OOP a standards-based place where the data produced on operator, edge and t
| Stakeholder | What they do | How they interact with the Data Lab |
| **Dataset contributor** (testbed / operator team) | Publishes datasets and keeps their descriptions current | UI registration or `PUT` on the Registration API with an API key; edits own records |
| **Dataset contributor** (testbed / operator team) | Publishes datasets and keeps their descriptions current | UI registration, or `PUT` on the Registration API with a client-credentials token; edits own records |
| **Experimenter / data consumer** | Finds and evaluates datasets for experiments or model training | Browses/searches the UI, reads quality results, follows access/download URLs, uses the Search API |
| **Data steward / operator** | Creates and retires catalogues (public or private), manages who may read and write each, promotes records from staging, manages vocabularies and API keys | Admin UI / Data Lab API (catalogues and membership); Registration API; Keycloak realm administration |
| **Data steward / operator** | Creates and retires catalogues (public or private), manages who may read and write each, promotes records from staging, creates the Keycloak clients of ingestion scripts, manages controlled vocabularies (*planned* — today they are fixed in the Data Lab API) | Admin UI / Data Lab API (catalogues and membership); Registration API; Keycloak realm administration |
| **Catalogue member** (reader, contributor or publisher of a catalogue) | A reader reads the catalogue's private datasets; a contributor also creates and edits its records; a publisher also deletes records and manages the catalogue's members | UI registration into the catalogues they are a contributor or publisher of; admin UI for that catalogue's members (publishers only) |
| **Ingestion pipeline** | Registers or synchronises records programmatically from partner systems | Registration API with an API key |
| **Ingestion pipeline** | Registers or synchronises records programmatically from partner systems | Registration API with a client-credentials token of its own Keycloak client |
| **Federation peer / data-space participant***(future)* | Discovers and negotiates access to assets across nodes | Dataspace Protocol via the connector (§H.6) |
---
@@ -149,15 +153,17 @@ Give OOP a standards-based place where the data produced on operator, edge and t
**Trigger:**`PUT /data-lab/hub/datasets/{dataset-id}?catalogue=oop-org` with a Turtle or JSON-LD body and an API key.
**Trigger:**`PUT /data-lab/hub/datasets/{dataset-id}?catalogue=oop-org&accessLevel={level}` with a Turtle or JSON-LD body and a bearer token the script obtained with the client-credentials grant of its own Keycloak client.
**Flow:**
1. Nginx routes the request to `piveau-hub-repo`.
2. The hub authenticates the API key and checks the caller may write to the target catalogue.
3.The record is validated against the loaded profile (DCAT-AP 3.0 + `dali-base` shapes). Violations return `400` with the SHACL report.
2. The hub validates the token and checks, through piveau access control, that the script's service account may write to the target catalogue (it is a contributor or publisher there, §H.7).
3.*Planned, not yet built:* the record is validated against the loaded profile (DCAT-AP 3.0 + `dali-base` shapes) and violations return `400` with the SHACL report. Today the hub checks RDF syntax only (F3).
4. The hub mints the dataset, distribution and catalogue-record URIs per the platform URI scheme (§H.2), persists the graph in Virtuoso, and writes the search document to Elasticsearch.
5.`201 Created` (or `200` on update). The dataset is immediately visible in search to everyone allowed to see it.
5.`201 Created` (or `200` on update). The dataset is visible in search to everyone allowed to see it.
**First version:** the record gets the `accessLevel` the script passes. The rule that a dataset is at least as strict as its catalogue (§H.7) is applied by the Data Lab API, not by the hub, so a script writing into a private catalogue must pass `internal` itself. A later version applies the rule and MAP validation to every write.
**Priority:** Must — **Reference:** MAP §5.1, §5.6, §11.1
@@ -277,7 +283,7 @@ The Data Lab sits beside the OOP exposure/fulfilment pipeline, not inside it:
| E3 | piveau-hub images are pinned (`piveau-hub-repo` 4.4.x from Docker Hub, `piveau-hub-search` 5.3.11 on Elasticsearch 8.13.x) with access control v2 enabled, and the platform Keycloak is 26.4 or newer. | Access control does not initialise (the hub logs `Failed to create Keycloak groups…`) and nothing is authorised; or the 6G-DALI baseline (hub-repo 3.3.x) differs in profile, auth or URI behaviour. Medium risk — mitigated by the smoke test (`scripts/smoke-test.sh`), which fails unless a private dataset is hidden from anonymous callers. |
| E4 | Dataset files are hosted elsewhere (partner storage, public URLs) and are reachable at the `dcat:accessURL` / `dcat:downloadURL` recorded. | Catalogue entries point at dead links; no data reachable. Medium risk — mitigated by the future Data Lab storage. |
| E5 | A single-node Virtuoso + Elasticsearch deployment on persistent volumes is acceptable for release volumes (§A.2). | Performance or durability incidents under load. Low risk at consortium scale. |
| E6 | Contributors obtain API keys / realm accounts out-of-band from the steward; no self-registration flow is needed for the release. | Onboarding friction. Low risk. |
| E6 | Contributors obtain realm accounts, and their scripts a Keycloak client, out-of-band from the steward; no self-registration flow is needed for the release. | Onboarding friction. Low risk. |
---
@@ -286,7 +292,7 @@ The Data Lab sits beside the OOP exposure/fulfilment pipeline, not inside it:
These are not resolved by the current text. Each carries a recommendation so implementers are not blocked.
-**F1. Ingress path and hostname.**`/data-lab/*` on the platform hostname vs. a dedicated `data.<platform-domain>` vhost. The Data Lab UI is hash-routed with relative asset paths, so it works under a prefix as well as at a root; piveau-hub assumes a base URL. *Recommendation:* path prefix `/data-lab/` with sub-paths `/data-lab/` (UI), `/data-lab/hub/` (repo API), `/data-lab/search/` (search API), `/data-lab/api/` (Data Lab API); no SPARQL route, and `/data-lab/admin/` only redirects to the UI's `#/admin`; set `PIVEAU_DCATAP_SCHEMA_CONFIG.baseUri` to the public base.
-**F2. Realm and role mapping.***Decided and tested end to end* (hub-repo 4.4.3, hub-search 5.3.11, Keycloak 26.4). Authorisation is piveau's access control v2, not a rule of ours: the hub creates, for every catalogue, a Keycloak group with Viewer / Expert / Publisher role groups and a Keycloak resource, and the members of **Piveau System › role: Operator** are the stewards. A Data Lab catalogue member is in all three role groups. The earlier group-named-after-the-catalogue rule and the `dl-steward` realm role are retired. Still to confirm: the realm name `dataspace`, and who is put in the Operator group in the platform realm (done by hand in Keycloak; the group exists only once the hub has started). Access control must be enabled when the deployment is first created: piveau does not support switching it on over existing data, and the Keycloak server must be 26.4 or newer (older ones reject the groups piveau creates).
-**F2. Realm and role mapping.***Decided and tested end to end* (hub-repo 4.4.3, hub-search 5.3.11, Keycloak 26.4). Authorisation is piveau's access control v2, not a rule of ours: the hub creates, for every catalogue, a Keycloak group with Viewer / Expert / Publisher role groups and a Keycloak resource, and the members of **Piveau System › role: Operator** are the stewards. A Data Lab catalogue member is in all three role groups. The earlier group-named-after-the-catalogue rule and the `dl-steward` realm role are retired. The realm name `dataspace` is fixed (ADR-0049). Still to confirm: who is put in the Operator group in the platform realm (done by hand in Keycloak; the group exists only once the hub has started). Access control must be enabled when the deployment is first created: piveau does not support switching it on over existing data, and the Keycloak server must be 26.4 or newer (older ones reject the groups piveau creates).
-**F3. Where MAP validation runs.** The deployed piveau instance loads a piveau *form/search profile* (`dcap-ap-simple/`, `piveau:mapping*` annotations) rather than the `dali-*` SHACL shapes. Options: (a) rely on piveau's built-in DCAT-AP validation plus the form profile; (b) run the layered `dali-*` shapes in `piveau-hub-repo`'s validation hook; (c) an external validator (pySHACL) in front of the Registration API. *Tested 2026-10-12 (hub-repo 3.3.11, hub-search 5.3.1; not repeated on the 4.4.3 / 5.3.11 pair):* option (a) does not work as assumed — with piveau profile mode on, hub-search answers 404 to the hub's index request and nothing is searchable; with it off (the 6G-DALI production setting) an empty `dcat:Dataset` is accepted with 201, so **no field is validated**. *Update:* records created or edited through the Data Lab UI/API are now validated against the MAP by the API (option (c), REQ-DL-22); records PUT straight to the hub's Registration API still are not. *Recommendation:* run with profile mode off for the release, state plainly that records are not validated, and make (b) or (c) the next step — it is what turns REQ-DL-03 from unmet to met.
-**F4. `dqv:QualityMeasurement` as a first-class node.** piveau-hub has no built-in notion of DQV measurements (MAP §11.6); they are stored as plain triples. Confirm they survive the repo's graph normalisation and that the UI renders them from the dataset graph. *Recommendation:* verify on the pinned image; fall back to embedding measurements as blank nodes if named-node measurements are dropped.
-**F5. Linking to future connector assets.**`dali:assetId` / `dali:connectorType` already anticipate EDC assets. Whether the EDC catalogue is generated *from* piveau records or registered *alongside* them is a next-release decision.
@@ -303,7 +309,7 @@ These are not resolved by the current text. Each carries a recommendation so imp
|----|-------------|----------|
| REQ-DL-01 | The system shall persist catalogues, datasets, distributions and catalogue records as RDF conforming to DCAT-AP 3.0 and the 6G-DALI MAP, in a triple store that is the source of truth. | MAP §1, §5, §8 |
| REQ-DL-02 | The system shall accept dataset and catalogue registrations over HTTP (`PUT`/`POST`/`DELETE`) with `text/turtle` and `application/ld+json` bodies (Registration API). | MAP §11.1 |
| REQ-DL-03 | The system shall validate every registration against the loaded profile and reject non-conformant records with a machine-readable validation report; mandatory MAP fields (§9.3) shall be enforced at minimum. | MAP §9.3, §11.7 |
| REQ-DL-03 | The system shall validate every registration against the loaded profile and reject non-conformant records with a machine-readable validation report; mandatory MAP fields (§9.3) shall be enforced at minimum. *Met through the Data Lab API (REQ-DL-22); planned, not yet met, for direct Registration API writes (F3).*| MAP §9.3, §11.7 |
| REQ-DL-04 | The system shall mint resource URIs under the platform base URI using the configured catalogue / dataset / distribution / record contexts (§H.2). | MAP §11.3, §12 |
| REQ-DL-05 | The system shall index every persisted record for keyword and faceted search (catalogue, theme, format, publisher, licence, keyword) and expose it over a REST Search API. | — |
| REQ-DL-06 | The triple store's SPARQL endpoint shall not be reachable from outside the compose network, because it holds private and public metadata together; public discovery shall use the Search API and the Data Lab API. | MAP §3 (deviation), §H.5 |
@@ -311,20 +317,20 @@ These are not resolved by the current text. Each carries a recommendation so imp
| REQ-DL-08 | The Data Lab UI shall open on the list of catalogues, list them, search datasets with facets on a separate Datasets page, and render a dataset page with core fields, testbed context, provenance, every distribution (format, size, checksum, access/download URL, measured variables) and the distribution's `dqv:QualityMeasurement`s, without requiring sign-in. | MAP §5.3–5.6 |
| REQ-DL-09 | The web UI shall treat a distribution with `dali:connectorType` set as "access via connector" and shall only offer `dcat:downloadURL` as a direct download. | MAP §5.6 |
| REQ-DL-10 | The Data Lab UI shall take its Keycloak and API settings from the container environment at start, without an image rebuild; branding (logo, title) is part of the UI source. | — |
| REQ-DL-11 | Interactive users shall authenticate with OIDC Authorization Code + PKCE against the platform Keycloak realm `dataspace`; machine clients shall authenticate with a per-client API key or a client-credentials token from the same realm. | ADR-0049 |
| REQ-DL-11 | Interactive users shall authenticate with OIDC Authorization Code + PKCE against the platform Keycloak realm `dataspace`; machine clients shall authenticate with a client-credentials token of their own Keycloak client in the same realm, with rights from their service account's catalogue membership. The hub API key shall be held by the Data Lab API only. | ADR-0049 |
| REQ-DL-12 | Anonymous read access to public catalogues, their datasets and search shall be allowed; all write operations shall require authentication and catalogue-level authorisation. | — |
| REQ-DL-13 | All Data Lab services shall be reachable only through the platform Nginx under `/data-lab/*`; no service shall be published on a host port in the target deployment. | ADR-0049 |
| REQ-DL-14 | The Data Lab shall not publish or consume DataBus subjects and shall not call OEG, SRM or FM internal APIs, nor read their databases. | ADR-0049, ADR-0025 |
| REQ-DL-15 | Each service shall expose a liveness/readiness check usable by Compose/Kubernetes health probes; the hub's readiness shall depend on the triple store and search service. | Platform baseline |
| REQ-DL-15 | Each service shall expose a liveness/readiness check usable by Compose/Kubernetes health probes; the hub's readiness shall depend on the triple store and search service. *The dependency-aware readiness is planned: today every probe checks only that the service itself answers.*| Platform baseline |
| REQ-DL-16 | The triple store and search index shall reside on persistent volumes; the search index shall be rebuildable from the triple store by an operator command. | §A.2 |
| REQ-DL-17 | Access to a catalogue shall be piveau's access control, offered at three levels: a *reader* (Viewer role group) reads its private datasets, a *contributor* (+ Expert) also creates and edits datasets, a *publisher* (+ Publisher) also deletes datasets and manages the members; the Data Lab API shall put users into and take them out of those role groups, which piveau creates with the catalogue, and shall not create or delete the groups itself. | §H.7 |
| REQ-DL-18 | Catalogue creation and deletion shall require membership of piveau's global Operator group (`catalog:create` on the system resource); managing a catalogue's members shall require that or the publisher level in the catalogue; listing in the management console shall show a non-steward only the catalogues they can write to. The Data Lab API shall derive these rights from the caller's permissions in Keycloak, not from claims it keeps itself. | §H.7 |
| REQ-DL-19 | The Data Lab API shall fail closed when Keycloak is not configured, and shall be the only component holding the Keycloak admin credential; its callers present bearer tokens only. | §H.5 |
| REQ-DL-20 | Deleting a non-empty catalogue shall be refused unless explicitly forced. | §H.7 |
| REQ-DL-21 | The datasets of a private catalogue, and of the staging catalogue, shall be registered with piveau access level `internal` (and any dataset at least as strictly as its own access rights ask: Public `public`, Restricted `restricted`, Non-public `internal`), so that no endpoint of piveau or of the Data Lab API reveals them to a caller without rights (listing, search, facets, counts or direct read); a read of a hidden dataset shall be indistinguishable from "not found". | §H.7 |
| REQ-DL-21 | The datasets of a private catalogue, and of the staging catalogue, shall be registered with piveau access level `internal` (and any dataset at least as strictly as its own access rights ask: Public `public`, Restricted `restricted`, Non-public `internal`), so that no endpoint of piveau or of the Data Lab API reveals them to a caller without rights (listing, search, facets, counts or direct read); a read of a hidden dataset shall be indistinguishable from "not found". *First version: this holds for datasets written through the Data Lab API; a direct Registration API write carries the access level its caller passes (UC-DL-01).*| §H.7 |
| REQ-DL-24 | For a signed-in user the Data Lab API shall extend the public view with exactly the catalogues the user is a member of at any level (every catalogue for a steward) — in the catalogue list, the search and its facets, and dataset reads — and shall answer "not found" for any other non-public catalogue; the Data Lab UI shall show them on the home page, marked with the caller's level and, where applicable, not public, and open them like public catalogues. | §H.7 |
| REQ-DL-22 | A dataset created or edited through the Data Lab API shall be validated against the MAP (mandatory fields, controlled values, GDPR/FAIR declarations, at least one distribution with title and access URL) and rejected with per-field errors; an edit shall not change the identifier and shall keep existing quality measurements. | MAP §5.1, §5.6, §12 |
| REQ-DL-23 | Deleting a dataset shall require membership of the Operator group or the publisher level in its catalogue; promoting a dataset from staging to production shall require the Operator group and shall register it with access level `public`; a failed promotion shall leave the dataset in staging. | §H.7 |
| REQ-DL-23 | Deleting a dataset shall require membership of the Operator group or the publisher level in its catalogue; promoting a dataset from staging to production shall require the Operator group and shall register it with the access level REQ-DL-27 gives it in production; a failed promotion shall leave the dataset in staging. | §H.7 |
| REQ-DL-27 | A dataset's access rights (Public, Restricted, Non-public) shall set its piveau access level (`public`, `restricted`, `internal`), and the level applied shall be the stricter of the catalogue's and the dataset's; changing the access rights of a dataset shall change who can see it. | §H.7 |
| REQ-DL-25 | A catalogue shall be created public or private, recorded as `dct:accessRights``PUBLIC` or `NON_PUBLIC`; the choice shall not be changeable afterwards, because it fixes the access level of every dataset written into the catalogue. | §H.7 |
| REQ-DL-26 | piveau access control v2 shall be enabled in hub-repo and hub-search from the first start of a deployment, on a Keycloak server of version 26.4 or newer; the realm shall give `piveau-hub-repo` authorization services, its `uma_protection` role and the realm-management roles it needs. | §H.1, §H.7 |
@@ -365,15 +371,15 @@ The normative model is the vendored MAP ([VERSION.md](/external-specs/6gdali-met
| Catalogue record | `dcat:CatalogRecord` | Minted by the hub; registration provenance |
| *(future)* Data service, ML model | `dcat:DataService`, `mldcat:MLModel` | Shapes vendored; not exercised in this release |
**URI scheme** (`PIVEAU_DCATAP_SCHEMA_CONFIG`), base URI = public Data Lab base (F1):
**URI scheme** (`PIVEAU_DCATAP_SCHEMA_CONFIG`), base URI = public Data Lab base (F1), ending in `/` (for example `https://oop.example.org/data-lab/`):
| distribution | `{base}set/distribution/{distribution-id}` |
| record | `{base}set/record/{dataset-id}` |
| metrics / history | `{base}id/metrics/…`, `{base}id/history_metrics/…` (shadow store) |
`dct:identifier` on the dataset carries the contributor's stable identifier (UUID/DOI) and must not change across re-registrations; the hub URI is derived from `{dataset-id}` in the request path. Persistence rules follow MAP §12.
@@ -381,8 +387,8 @@ The normative model is the vendored MAP ([VERSION.md](/external-specs/6gdali-met
| **Registration & catalogue API** | `piveau-hub-repo` | `/data-lab/hub/datasets/{id}?catalogue={cat}`, `/data-lab/hub/catalogues/{id}` — `PUT`/`GET`/`DELETE`, `text/turtle` / `application/ld+json` | Bearer token (realm `dataspace`; scripts use client credentials); anonymous `GET` of what is public. The API key (`X-API-Key`) is used by the Data Lab API only | Contributor scripts, ingestion pipelines, Data Lab API |
| **Search API** | `piveau-hub-search` | `/data-lab/search/datasets?q=…&facets=…`, `/data-lab/search/catalogues` | Anonymous read; a bearer token widens it to the caller's own catalogues | Data Lab API, consumers |
| **SPARQL endpoint** | Virtuoso (internal) | not routed by Nginx; reachable inside the compose network only (dev overlay publishes it for debugging) | — | operators |
| **Data Lab API** | `data-lab-api` | `/data-lab/api/` — *anonymous:*`GET /public/catalogues`, `GET /public/datasets?q=&catalog=&categories=&format=&license=&keywords=&publisher=&sort=&page=`, `GET /public/datasets/{catalogue}/{id}`, `GET /vocab`, `GET /health`. *Bearer token:*`GET /browse/catalogues` (public + the caller's own, flagged `public` and with the caller's `level`; `POST /catalogues` takes `visibility``public`|`private`), `GET /browse/datasets?…` and `GET /browse/datasets/{catalogue}/{id}` (same shape as `/public/*`, extended to the caller's own catalogues), `GET/POST /catalogues`, `PUT/DELETE /catalogues/{id}`, `GET/POST /catalogues/{id}/members` (a member is added with a `level`), `PUT/DELETE /catalogues/{id}/members/{user}` (change level / remove), `GET/POST /catalogues/{id}/datasets`, `GET/PUT/DELETE /catalogues/{id}/datasets/{ds}`, `POST …/datasets/{ds}/promote`, `GET /users?q=`, `GET /me` (includes the caller's level per catalogue) | Anonymous read; bearer token (realm `dataspace`) for the rest | Data Lab UI; DataOps orchestrator (future) |
| **UI** | `data-lab-admin-ui` (static) | `/data-lab/` (hash routes: `#/` catalogue list, `#/datasets` dataset search, `#/catalogues/{id}`, `#/datasets/{cat}/{id}`, `#/admin…`); its own `/api/` is proxied to `data-lab-api` | Anonymous for public pages; OIDC (PKCE) for `#/admin` | Everyone; stewards and catalogue contributors and publishers for the console |
@@ -392,21 +398,21 @@ No interface of the Data Lab is part of OEG's NBI; none is CAMARA/TMF-shaped. Er
### H.4 Deployment
-**Compose profile `data-lab`** (added to the root `docker-compose.yml`): the seven services in §H.1 on the platform network; depends on profile `core` for Nginx and Keycloak only.
-**Compose profile `data-lab`** (`compose/docker-compose.yml` in the Data Lab repo, its own compose project `oop-data-lab`): the seven services in §H.1 on their own `data-lab` network, which the platform Nginx joins to reach them; depends on profile `core` for Nginx and Keycloak only.
-**Nginx:** route family `/data-lab/*` → UI / hub / search / API with prefix stripping; no SPARQL route; forward `Authorization`.
-**Keycloak:** realm `dataspace` (Keycloak 26.4 or newer) provisioned from a realm export checked into the Data Lab repo: client `data-lab-admin` (public/PKCE, the UI), `piveau-hub-repo` (confidential; authorization services with its own `uma_protection` role; service account with the `realm-management` roles `manage-users`, `view-users`, `query-users`, `query-groups`, `manage-authorization`, `manage-clients`, `view-clients`, `query-clients`, `view-realm`, `view-authorization`) and `data-lab-api` (confidential; service account with `manage-users`, `query-users`, `query-groups`, `view-users`). The catalogue groups and the Operator group are created by piveau at runtime; stewards are added to **Piveau System › role: Operator** by an administrator. The UI container needs `KEYCLOAK_SERVER_URL`, `KEYCLOAK_REALM`, the client id and the Keycloak *origin* (for its CSP) at start.
-**Configuration keys** (hub): `PIVEAU_HUB_API_KEY`, `PIVEAU_HUB_SEARCH_SERVICE`, `PIVEAU_TRIPLESTORE_CONFIG`, `PIVEAU_SHADOW_TRIPLESTORE_CONFIG`, `PIVEAU_HUB_AUTHORIZATION_PROCESS_DATA` (issuer = platform Keycloak realm URL), `PIVEAU_HUB_CORS_DOMAINS` (platform origin only — not `*`), `PIVEAU_PROFILE` / `PIVEAU_FEATURE_FLAGS`, `PIVEAU_DCATAP_SCHEMA_CONFIG`. Secrets are injected from the environment / Kubernetes Secrets, never committed.
-**UI:**`VITE_*` branding and base URLs set at container start (title "OOP Data Lab", platform logo/palette, hub/search/Keycloak URLs).
-**UI:**branding (title "OOP Data Lab", platform logo) is built into the image (REQ-DL-10). At container start `config.js` is rendered from `KEYCLOAK_SERVER_URL`, `KEYCLOAK_REALM` and `KEYCLOAK_ADMIN_UI_CLIENT_ID`; `KEYCLOAK_ORIGIN` goes into the CSP and `DATA_LAB_API_UPSTREAM` is where `/api/` is proxied.
-**Kubernetes:** same seven workloads as a Helm chart in a later step; no requirement for the release beyond Compose.
-**Kubernetes (alternative):** subchart `datalab` of the OOP Helm chart, off by default: one Virtuoso, the UI as the only public entry on a NodePort, hub and search cluster-internal, realm `dataspace` provisioned by the OOP Keycloak chart. The OOP Helm chart has no Nginx, so REQ-DL-13 applies to the compose profile; one Virtuoso instead of the two of §H.1 (ADR-0069).
### H.5 Security
| Boundary | Mechanism |
|---|---|
| User → Nginx → UI | HTTPS; OIDC Authorization Code + PKCE (realm `dataspace`); tokens held in the browser SPA (piveau.kit default) |
| Script → Nginx → hub | HTTPS; `X-API-Key` per client, or client-credentials bearer token |
| UI → hub / search | Bearer token forwarded; CORS restricted to the platform origin |
| User → Nginx → UI | HTTPS; OIDC Authorization Code + PKCE (realm `dataspace`); tokens held in the browser by the SPA (`keycloak-js`) |
| Script → Nginx → hub | HTTPS; client-credentials bearer token of the script's own Keycloak client; its rights are its service account's catalogue memberships. Scripts never get the hub API key |
| Data Lab API → hub / search (reads) | The caller's bearer token is forwarded, so piveau filters by the caller's rights; the hub's CORS is restricted to the platform origin |
| hub → Virtuoso / Elasticsearch | Cluster network only; Virtuoso `dba` password and ES credentials from secrets |
| Admin UI → Data Lab API | Bearer token; the API re-checks role and group on every call (the UI hides, the API enforces); same-origin via the UI container's `/api/` proxy, so no CORS; the UI is a separate static app with a CSP of `default-src 'self'` plus the Keycloak origin, no inline script, all assets bundled (no third-party CDN); user-controlled text is rendered through React, never as HTML; catalogue ids restricted to `[a-z0-9-]` (they become a URI segment, a group name and a path) |
| Data Lab API → Keycloak / hub | Service-account client credentials and hub API key held only by the API container, from secrets. Catalogue role groups carry no administrative scope, so membership of a catalogue can never escalate to steward |
@@ -430,7 +436,7 @@ Data-protection rules: metadata records must not contain subscriber identifiers
**The rule.** Access is **piveau's access control v2**, not a rule of ours. When a catalogue is registered, piveau creates a Keycloak group for it (named after the catalogue title, with the catalogue id as attribute) holding the role groups *Data Viewer* (`dataset:view_published`), *Data Expert* (`dataset:create`, `dataset:update`, `dataset:view_draft`) and *Data Publisher* (`dataset:publish`, `dataset:delete`), plus a Keycloak resource named after the catalogue id. A **Data Lab catalogue member** has one of three levels, each a set of those role groups: *reader* (Viewer) reads the catalogue's private datasets; *contributor* (Viewer + Expert) also creates and edits datasets; *publisher* (all three) also deletes datasets and manages the members. A **steward** is a member of the global group *Piveau System › role: Operator*, which holds every scope on the system resource `0_global`. Anyone else sees public data only. Nothing in the Data Lab keeps a parallel list of who may do what.
**Public and private.** Every catalogue is created public or private; the choice is recorded on the catalogue as `dct:accessRights` (`PUBLIC` or `NON_PUBLIC`) and cannot be changed afterwards. The staging catalogue is private by definition. The API writes every dataset of a private catalogue with piveau access level `internal` and every dataset of a public one with `public`; **piveau then enforces it by itself** — `piveau-hub-repo` answers 403 and `piveau-hub-search` omits the dataset (including from facets) for a caller without rights, whichever route they use. Catalogue records are exempt from access control in piveau, so a private catalogue's title and description remain visible to anyone who can reach the hub; its datasets and their count do not. Promotion to a public catalogue re-registers the dataset with the level its own access rights ask for.
**Public and private.** Every catalogue is created public or private; the choice is recorded on the catalogue as `dct:accessRights` (`PUBLIC` or `NON_PUBLIC`) and cannot be changed afterwards. The staging catalogue is private by definition. The API writes every dataset of a private catalogue with piveau access level `internal`, and every dataset of a public one with the level its own access rights ask for (below); **piveau then enforces it by itself** — `piveau-hub-repo` answers 403 and `piveau-hub-search` omits the dataset (including from facets) for a caller without rights, whichever route they use. Catalogue records are exempt from access control in piveau, so a private catalogue's title and description remain visible to anyone who can reach the hub; its datasets and their count do not. Promotion to a public catalogue re-registers the dataset with the level its own access rights ask for.
**Access rights of a dataset.** The *Access rights* field of the MAP form (`dct:accessRights`: Public, Restricted, Non-public) is not only a description: the API maps it onto piveau's three access levels when it writes the record, and piveau enforces them. The level applied is the **stricter of the catalogue's and the dataset's**, so a dataset can be made stricter than its catalogue but a private catalogue can never leak a dataset into public view. Changing a dataset's level is therefore an edit of that field.
@@ -453,17 +459,17 @@ The mapping is ours: piveau's names and the DCAT-AP access-right terms do not me
| Delete catalogue | steward | Refused for bootstrap catalogues; refused if non-empty (or if emptiness cannot be determined) unless `force`; then delete in the hub, and piveau removes the Keycloak groups and resource itself |
| List catalogues (console) | any signed-in user | Stewards see all; others see the catalogues they can write to, with the count of datasets they may see |
| List / add / remove members, change a member's level | steward, or a publisher of that catalogue | Membership of the catalogue's Viewer, Expert and Publisher role groups, set by level; users are found by name, username or e-mail in the realm |
| Create / edit a dataset | steward, contributor or publisher | Validated against the MAP, written with the catalogue's access level |
| Create / edit a dataset | steward, contributor or publisher | Validated against the MAP, written with the stricter of the catalogue's and the dataset's access level |
| Delete a dataset | steward or publisher | |
| Browse / search / read datasets | anyone (public) or signed in (`/browse/*`) | Read as the caller; piveau decides what is visible |
**Why a separate UI app.** The UI holds no secret and no rule: it is a client of the API like any other, so it ships, scales and is secured as a static site, and it can later be folded into the DataOps UI (also React) or replaced without touching the API. Authorisation lives only in the API.
**Why a separate API.** Creating a catalogue and wiring its access is two systems (piveau, Keycloak) that have to change together, and both need credentials that browsers and other services must not hold. The API owns that pair, keeps the rule in one place, and is the stable surface the DataOps services call when they register testbeds (F8), so no second component grows its own Keycloak/piveau client. It is intentionally small — catalogue lifecycle and membership only; dataset registration stays on the piveau Registration API, and the API adds no business logic to metadata.
**Why a separate API.** Creating a catalogue and wiring its access is two systems (piveau, Keycloak) that have to change together, and both need credentials that browsers and other services must not hold. The API owns that pair, keeps the rule in one place, and is the stable surface the DataOps services call when they register testbeds (F8), so no second component grows its own Keycloak/piveau client. It is intentionally small: catalogue lifecycle, membership, and dataset create / edit / delete / promote with MAP validation (UC-DL-02, UC-DL-08). Scripts may still register datasets on the piveau Registration API directly (UC-DL-01).
**Failure model.** Keycloak unconfigured → the API answers 503 and serves nothing (fail closed). Keycloak or piveau unreachable → 502 with the upstream status. If piveau access control cannot initialise (old Keycloak, missing service-account roles) the hub logs it and catalogues get no groups: nobody but the API key holder can then write, which is safe but unusable, and `add member` answers 409 naming the cause. Membership changes take effect within about 30 seconds (the permissions cache), with no new login.
**Known limits.** No audit trail beyond Nginx and application logs in this release; three fixed levels (piveau's role configuration could define others); a catalogue's visibility cannot be changed after creation; no cross-realm users; piveau cannot be switched to access control v2 over existing data.
**Known limits.**First version: the access-level rule above and MAP validation are applied by the Data Lab API only. A record PUT straight to the hub's Registration API carries the access level its caller passes and is not validated (UC-DL-01); a later version applies both to every write. No audit trail beyond Nginx and application logs in this release; three fixed levels (piveau's role configuration could define others); a catalogue's visibility cannot be changed after creation; no cross-realm users; piveau cannot be switched to access control v2 over existing data.
---
@@ -471,10 +477,12 @@ The mapping is ours: piveau's names and the DCAT-AP access-right terms do not me
### Structured Logging
piveau services are Java/Vert.x and do not use `oop-commons`. They shall log JSON to stdout at `INFO` in the target deployment (the 6G-DALI baseline's `TRACE` level is for development only), and Nginx access logs carry the request id that the hub echoes, so a registration can be traced end-to-end. The UI logs nothing server-side.
piveau services are Java/Vert.x and do not use `oop-commons`. They log at `INFO` in a deployment (the 6G-DALI baseline's `TRACE` level is for development only). *Planned, not yet configured:* JSON log output, and a request id carried from the Nginx access logs through the hub, so a registration can be traced end-to-end. The UI logs nothing server-side.
### Key Metrics
*Planned, not yet built:* the Data Lab emits none of these metrics yet. They are the target set.
| Metric | Type | Description |
|---|---|---|
| `datalab_registrations_total` | Counter | Registration API writes, labelled by catalogue and outcome (created / updated / rejected) |
@@ -496,6 +504,8 @@ Sourced from Nginx access logs and the hub's metrics until native Prometheus exp
The dependency-aware readiness of `piveau-hub-repo` and `piveau-hub-search` is planned (REQ-DL-15). Today their readiness is the same HTTP 200 on `/` as their liveness.
### Operations
-**Backup:** nightly Virtuoso dump (both instances) to the platform backup target; Elasticsearch is not backed up — re-index from the triple store with the hub's re-index command.
@@ -520,3 +530,4 @@ Terminology is owned by the single platform glossary — see [platform/glossary.
| 0.1 (Draft) | 2026-10-13 | Private catalogues through piveau access control v2 (hub-repo 4.4.3, hub-search 5.3.11, Keycloak 26.4+): per-dataset access level, Viewer / Expert / Publisher role groups per catalogue and an Operator group for stewards replace the group-named-after-the-catalogue rule and the `dl-steward` role; public or private chosen at catalogue creation; staging is private; the public SPARQL endpoint is withdrawn (UC-DL-05, REQ-DL-06/17/18/21/23/25/26, F2, §H.5, §H.7). Tested end to end. |
| 0.1 (Draft) | 2026-10-13 | Members are added at a level — reader, contributor or publisher — mapped onto piveau's Viewer / Expert / Publisher role groups, and the level can be changed; deleting datasets and managing members need the publisher level (UC-DL-07, REQ-DL-17/18/23, §H.7). |
| 0.1 (Draft) | 2026-10-13 | A dataset's access rights (Public / Restricted / Non-public) now set its piveau access level (public / restricted / internal), the stricter of the catalogue's and the dataset's applying; moving a dataset between levels is an edit of that field (§H.7, REQ-DL-21, REQ-DL-27). |
| 0.1 (Draft) | 2026-10-16 | Scripts authenticate with a client-credentials token of their own Keycloak client; the hub API key stays inside the Data Lab API (UC-DL-01, B, E6, REQ-DL-11, §H.3, §H.5). First version: direct Registration API writes are not MAP-validated and carry the access level their caller passes (UC-DL-01, REQ-DL-03/21, §H.7). Text aligned with the code: diagram, §H.2 URI templates, §H.4 compose and UI configuration, §H.5, §H.7, REQ-DL-23, F2. Metrics, structured logging, dependency-aware readiness and vocabulary management marked as planned (§I, REQ-DL-15, B). |
Changes for decisions/index.md: 1 added line, 0 removed lines.
Original line number
Diff line number
Diff line
@@ -52,3 +52,4 @@ the reasoning lives here. Never renumber; supersede instead (see the record-deci
*[ADR-0048](0048-data-lab-piveau-hub-and-6gdali-map.md) - Data Lab adopts piveau-hub as catalogue engine and the 6G-DALI Metadata Application Profile (DCAT-AP 3.0 + DQV + PROV-O) as normative metadata profile
*[ADR-0049](0049-data-lab-standalone-profile-shared-ingress-and-iam.md) - Data Lab is a standalone compose profile behind the platform Nginx and Keycloak (realm `dataspace`); no DataBus/SRM/OEG coupling in the catalogue release
*[ADR-0050](0050-data-lab-draft-version-not-pinned.md) - data-lab/ specs carry version 0.1 (Draft) and are excluded from the Release 2 pin (extends ADR-0021 exceptions)
*[ADR-0069](0069-data-lab-helm-chart-alternative-to-compose.md) - The Data Lab can also be deployed with the OOP Helm chart: off by default, UI on a NodePort, one Virtuoso, hub and search cluster-internal (proposed)
| Contributor / consumer → Nginx → Data Lab | OIDC Authorization Code + PKCE (Keycloak realm `dataspace`) for the UI; `X-API-Key` or client-credentials token for scripted registration; anonymous read (ADR-0049) | Data Lab |
| Contributor / consumer → Nginx → Data Lab | OIDC Authorization Code + PKCE (Keycloak realm `dataspace`) for the UI; client-credentials token of the script's own Keycloak client for scripted registration (the hub API key stays inside the Data Lab API); anonymous read (ADR-0049) | Data Lab |
### Roles
@@ -843,7 +843,7 @@ All components use `oop-commons` structured JSON logging. Every log entry includ
| FM | Process alive | PostgreSQL + DataBus connectivity |
| Portal BFF | Process alive | OEG reachable + IAM reachable |
| Data Lab (`piveau-hub-repo`) | HTTP 200 on `/` | Virtuoso + `piveau-hub-search` (Elasticsearch) reachable |
| Data Lab (`piveau-hub-repo`) | HTTP 200 on `/` | Virtuoso + `piveau-hub-search` (Elasticsearch) reachable (planned; today HTTP 200 on `/`) |
### Key Metrics (Phase 2)
@@ -1513,3 +1513,4 @@ Terminology is owned by the single platform glossary — see
| 2.0 | 2026-09-01 | Ownership Matrix corrected: RFC 7807 problem+json construction for SBI refusals is SRM's, not TF-SDK's. `can_serve_target` interface listing updated to the tri-state answer (ADR-0040) and TF-SDK-owned adapter factory (ADR-0041). |
| 2.0 | 2026-09-14 | Data Lab (draft component, next release) registered: component overview row, boundary-matrix rows for the dataset catalogue and the reserved data-transfer responsibility, security and health-check rows, compose profile `data-lab`, scaling row, Phase 4 deliverables 7–8, Intelligence-domain cross-reference (ADR-0048, ADR-0049, ADR-0050). |
| 2.0 | 2026-09-04 | Component Boundary Matrix updated: operator-triggered federation establishment is proxied through OEG (`command.fm.*`, ADR-0043); Edge Cloud Zone locality determination for deployment targets moves from OEG to SRM as an interim layer pending a future runtime-coordinator component (ADR-0044). Placement-decisions row absorbs the new locality row (was listing an inconsistent Not-Owner set); federation admin CRUD folded into the existing EWBI row instead of a redundant new one; Not-Owner columns corrected to name the passive consumer (FM) rather than an unrelated component. |
| 2.0 | 2026-10-16 | Data Lab security row: scripts register with a client-credentials token of their own Keycloak client; the hub API key stays inside the Data Lab API. Data Lab readiness row marked planned (data-lab architecture REQ-DL-15). |
Changes for platform/glossary.md: 2 added lines, 1 removed line.
Original line number
Diff line number
Diff line
@@ -157,6 +157,7 @@ This is the single glossary for all Open Operator Platform (OOP) architecture do
| 2.0 | 2026-07-29 | RFC 7807 entry corrected: no longer claims all OOP components use it. It is SRM's internal-API format only; OEG's northbound API uses CAMARA's `ErrorInfo` (ADR-0030). |
| 2.0 | 2026-08-05 | Capability instance entry corrected: `external_ref` is the backend realization ref, not a caller-facing id — caller-facing identity lives on `service_instance.id` for every family (ADR-0033). |
| 2.0 | 2026-09-01 | Placement, Network Domain, and Control path binding entries note the device-targeted control-path-probe exception (no zone/domain pin; adapter probing, ADR-0038/ADR-0040) and TF-SDK's adapter factory (ADR-0041). |
| 2.0 | 2026-10-16 | Staging catalogue entry corrected: records are promoted to the production catalogue `oop-org`, not to `6g-dali*` catalogues. |
## Data Lab Concepts
@@ -169,5 +170,5 @@ This is the single glossary for all Open Operator Platform (OOP) architecture do
| **DQV** | W3C Data Quality Vocabulary; `dqv:QualityMeasurement` records per distribution carry pass/fail results of data-quality expectations. |
| **PROV-O** | W3C provenance ontology; `prov:wasDerivedFrom` links derived datasets to their sources. |
| **SHACL** | W3C Shapes Constraint Language; the MAP's layered `dali-*` shapes validate records before registration. |
| **Staging catalogue** | The Data Lab catalogue holding submitted, not-yet-reviewed records; promotion to the community (`6g-dali*`) catalogues is a steward action. |
| **Staging catalogue** | The Data Lab catalogue holding submitted, not-yet-reviewed records; promotion to the production catalogue (`oop-org`) is a steward action. |
| **Dataspace Protocol (DSP) / EDC** | IDSA protocol for catalogue exchange, contract negotiation and transfer, and its Eclipse reference connector — the future Data Lab connector (not in the catalogue release). |