docs(data-lab): add Data Lab architecture spec and supporting ADRs

---

type: Implementation & Delivery

title: "Data Lab — Implementation & Delivery"

description: "How the Data Lab code is laid out, its settings, how it is built, deployed and operated, what it logs and probes, how it is tested, its known gaps, what comes next, and the changes developers make most often."

tags: [data-lab, implementation, delivery, operations]

timestamp: 2026-10-16T00:00:00Z

component: Data Lab (DL)

version: "0.1.0"

status: "Released — catalogue 0.1.0; implementation status in requirements-and-design.md §H.1"

---

# Data Lab — Implementation & Delivery

**Companion documents:**

[data-lab/overview-and-architecture.md](/data-lab/overview-and-architecture.md),

[data-lab/requirements-and-design.md](/data-lab/requirements-and-design.md),

[data-lab/interface-contract.md](/data-lab/interface-contract.md)

Code: the `data-lab` repository (Data Lab API, Data Lab UI, Compose stack, scripts) and the `helm`

repository (`oop-platform-chart/charts/datalab`, realm `dataspace` in `charts/keycloak`). This

document describes data-lab release `v0.1.0` and helm tag `datalab-0.1.0`.

---

## O. Code Layout

### O.1 The `data-lab` Repository

| Folder | Holds |

|---|---|

| `api/` | The Data Lab API (FastAPI) and its tests |

| `admin-ui/` | The Data Lab UI (React, TypeScript, Vite), its Nginx template and start-up script |

| `compose/` | `docker-compose.yml` (the stack), `docker-compose.dev.yml` (host ports and a local gateway), `docker-compose.local.yml` (a throwaway Keycloak) |

| `nginx/` | `data-lab.conf`, the `/data-lab/*` routes the platform Nginx includes; `dev-gateway.conf` for local use |

| `keycloak/` | `realm-dataspace.json`, the realm export to import into the platform Keycloak |

| `bootstrap/` | `bootstrap.sh` and the Turtle of the two bootstrap catalogues |

| `config/profile/` | piveau form and search profile; mounted but inactive ([requirements-and-design.md §L.1](/data-lab/requirements-and-design.md)) |

| `config/shapes/` | Vendored MAP SHACL shapes; staged, not loaded ([requirements-and-design.md §L.1](/data-lab/requirements-and-design.md)) |

| `scripts/` | `smoke-test.sh`, `reindex.sh` |

| `local/` | `local-up.sh`, `local-down.sh`, `e2e.py`, and the generator of the local-only realm |

| `examples/` | Sample MAP datasets in Turtle |

| `docs/` | `deployment.md`, `operations.md` |

### O.2 Data Lab API (`api/app/`)

| Module | Holds |

|---|---|

| `main.py` | Routes, request models, id checks |

| `auth.py` | Token checks, the permissions lookup and its cache, the route guards (signed in, contributor, publisher, steward) |

| `config.py` | Settings, read from the environment once at import |

| `hub.py` | Catalogue calls to `piveau-hub-repo` and `piveau-hub-search`; visibility and counts |

| `datasets.py` | Dataset list, read, write, delete and promote; access levels; parsing a record into the summary |

| `public.py` | The anonymous and signed-in reads: catalogue lists and faceted search |

| `mapform.py` | The vocabulary, MAP validation, and building the Turtle from the form |

| `keycloak_admin.py` | Membership through the Keycloak admin API |

### O.3 Data Lab UI (`admin-ui/src/`)

| Folder or file | Holds |

|---|---|

| `pages/` | Public pages: search, catalogues, dataset |

| `pages/admin/` | Console pages: home, dataset form |

| `components/` | Catalogue table, create-catalogue form, datasets panel, members panel, dataset detail and form views |

| `layout/` | Public and console page frames |

| `api.ts`, `http.ts` | Calls to the Data Lab API and error handling |

| `auth.ts`, `authContext.tsx` | Keycloak sign-in |

| `config.ts` | Runtime settings from `config.js` |

| `router.ts` | The hash router |

| `form.ts`, `search.ts`, `types.ts` | Form state, search parameters, types |

### O.4 Helm (`oop-platform-chart/charts/datalab/`)

`templates/` holds one manifest per workload (hub-repo, hub-search, Virtuoso, Elasticsearch, API,

UI), the Secret and the setup Job. `files/` holds the catalogue Turtle and `setup.py`, which the Job

runs. Realm `dataspace` is defined in `charts/keycloak`.

---

## P. Configuration

Compose reads `.env` (template: `.env.example`). Generate secrets per deployment and inject them

from the platform secret store; never commit them. Helm sets the same values from its own values

and Secrets (§Q.6).

| Group | Settings | Notes |

|---|---|---|

| Public identity | `DATA_LAB_BASE_URI`, `DATA_LAB_PUBLIC_ORIGIN` | Required. `DATA_LAB_BASE_URI` ends in `/` and mints every URI: set it before the first registration and never change it ([requirements-and-design.md §L.3](/data-lab/requirements-and-design.md)). `DATA_LAB_PUBLIC_ORIGIN` is the single browser origin the hub's CORS allows, never `*` |

| Catalogues | `CATALOGUE_ID` (`oop-org`), `CATALOGUE_STAGING_ID` (`oop-org-staging`), `CATALOGUE_TITLE` (`OOP Data Lab`), `CATALOGUE_PUBLISHER` (`OOP`) | Used by the bootstrap script and the API |

| Secrets | `PIVEAU_REPO_API_KEY`, `PIVEAU_SEARCH_API_KEY`, `VIRTUOSO_DBA_PASSWORD`, `KEYCLOAK_HUB_CLIENT_SECRET`, `KEYCLOAK_API_CLIENT_SECRET` | API keys: `openssl rand -hex 32`. Keycloak secrets: regenerate them in Keycloak. **Note:** `VIRTUOSO_DBA_PASSWORD` must be 16 characters or fewer (`openssl rand -hex 8`); Virtuoso 7.2.14 refuses longer ones and every hub write then fails with `401` |

| Keycloak | `KEYCLOAK_SERVER_URL`, `KEYCLOAK_REALM` (`dataspace`), `KEYCLOAK_ISSUER`, `KEYCLOAK_ORIGIN`, `KEYCLOAK_HUB_CLIENT_ID` (`piveau-hub-repo`), `KEYCLOAK_ADMIN_UI_CLIENT_ID` (`data-lab-admin`), `KEYCLOAK_API_CLIENT_ID` (`data-lab-api`) | Must match the platform Keycloak. `KEYCLOAK_ORIGIN` is scheme and host only, no path; it goes into the UI's Content Security Policy. The `.env.example` hostnames are deployment placeholders (F1) |

| Images | `PIVEAU_HUB_REPO_IMAGE`, `PIVEAU_HUB_SEARCH_IMAGE`, `VIRTUOSO_IMAGE`, `ELASTICSEARCH_IMAGE`, `DATA_LAB_API_IMAGE`, `DATA_LAB_UI_IMAGE` | Pinned, never `:latest` (§Q.1). The two Data Lab images are required; Compose refuses to start without them |

| Runtime | `COMPOSE_PROFILES` (`data-lab`), `PIVEAU_LOG_LEVEL` (`INFO`), `HUB_JAVA_OPTS` and `SEARCH_JAVA_OPTS` (`-Xms1g -Xmx2g`), `ES_JAVA_OPTS` (`-Xms2g -Xmx4g`) | **Note:** never `TRACE` in a deployment; it writes record contents to stdout |

| piveau hub | `PIVEAU_HUB_API_KEY`, `PIVEAU_HUB_SEARCH_SERVICE`, `PIVEAU_TRIPLESTORE_CONFIG`, `PIVEAU_SHADOW_TRIPLESTORE_CONFIG`, `PIVEAU_HUB_AUTHORIZATION_PROCESS_DATA`, `PIVEAU_HUB_CORS_DOMAINS`, `PIVEAU_PROFILE`, `PIVEAU_FEATURE_FLAGS`, `PIVEAU_DCATAP_SCHEMA_CONFIG` | Set in `docker-compose.yml` from the groups above. The authorization issuer is the realm URL. `PIVEAU_FEATURE_FLAGS` is `{"piveau_profile":false,"piveau_access_control_v2":true}` on both hub services; AC2 is fixed for the life of a deployment. The triple-store setting includes `clearGeoDataCatalogues: ["*"]`. **Note:** `PIVEAU_HUB_SEARCH_SERVICE` must have `"enabled":true`; with `false` (the 6G-DALI baseline) nothing is indexed and no error is logged |

| piveau search | `PIVEAU_HUB_SEARCH_API_KEY`, `PIVEAU_HUB_SEARCH_ES_CONFIG`, `PIVEAU_HUB_SEARCH_GAZETTEER_CONFIG` | The gazetteer is unused but must have a URL |

| Data Lab API | `KEYCLOAK_ISSUER`, `KEYCLOAK_INTERNAL_URL`, `KEYCLOAK_API_CLIENT_ID`, `KEYCLOAK_API_CLIENT_SECRET`, `KEYCLOAK_HUB_CLIENT_ID`, `PIVEAU_HUB_URL`, `PIVEAU_SEARCH_URL`, `PIVEAU_REPO_API_KEY`, `DATA_LAB_BASE_URI`, `DATA_LAB_PUBLIC_ORIGIN`, catalogue settings, `DL_PERMISSIONS_TTL` (`30` seconds) | `KEYCLOAK_ISSUER` empty: every signed-in route answers `503`. `KEYCLOAK_INTERNAL_URL` is optional: where the issuer is not reachable from the API (Kubernetes), the API calls Keycloak there and still checks tokens against the issuer. Missing hub key or base URI: hub calls answer `503`. Missing API client secret: membership calls answer `503` |

| Data Lab UI | `KEYCLOAK_SERVER_URL`, `KEYCLOAK_REALM`, `KEYCLOAK_ADMIN_UI_CLIENT_ID`, `KEYCLOAK_ORIGIN`, `DATA_LAB_API_UPSTREAM` (`http://data-lab-api:8000`), `DATA_LAB_API_URL` (browser-side API base, default `api`) | Rendered into `config.js` and the Nginx template at start (REQ-DL-10). The `/api/` proxy resolves the API through the container's own DNS servers, so it works under Compose and Kubernetes |

| Development | `DEV_PORT_*`, `LOCAL_PORT_BASE` (`11000`), `LOCAL_KC_PORT` (`8180`), `LOCAL_PROJECT` | Local use only (§Q.7) |

Unspecified: the API's timeouts (10–60 seconds per call today) and the 100-catalogue listing limit

are fixed in code, not settings. T32 records the optional settings not forwarded by base Compose.

---

## Q. Build and Deployment

### Q.1 Images

| Component | Image |

|---|---|

| `piveau-hub-repo` | `piveau/hub-repo:4.4.3` |

| `piveau-hub-search` | `registry.gitlab.com/piveau/hub/piveau-hub-search:5.3.11` |

| Virtuoso, shadow Virtuoso | `openlink/virtuoso-opensource-7:7.2.14-alpine` |

| Elasticsearch | `docker.elastic.co/elasticsearch/elasticsearch:8.13.4` |

| Data Lab API | Built from `api/` on `python:3.12-slim` |

| Data Lab UI | Built from `admin-ui/`: Node builds the app, `nginx:1.27-alpine` serves it |

Virtuoso stays on 7.2.14 rather than a newer 7.2.x because that version is proven against

`piveau-hub-repo` (E3). The 6G-DALI baseline's shadow store ran 7.2.8; that was drift, not a

requirement.

### Q.2 CI

`.gitlab-ci.yml` tests both Data Lab images (§S), checks the deployment files, and publishes the

images to the project's GitLab Container Registry with two tags:

| Tag | Meaning |

|---|---|

| `ci-<branch>-<short-sha>` | One per commit, never changes; pin this in a deployment |

| `<branch>` | Moves with every build of that branch |

`main` and `develop` publish automatically; any other branch publishes when the `build-and-push`

job is started by hand.

### Q.3 Compose Profile

`compose/docker-compose.yml`, project `oop-data-lab`, profile `data-lab`:

1. Seven services on their own `data-lab` network: Elasticsearch, Virtuoso, shadow Virtuoso,

   `piveau-hub-search`, `piveau-hub-repo`, `data-lab-api`, `data-lab-admin-ui`.

2. Start order: Elasticsearch, then hub-search; Virtuoso and the shadow store; then hub-repo once

   all three are healthy; then the API; then the UI.

3. Volumes: `virtuoso_db`, `shadow_virtuoso_db`, `elasticsearch`, plus the read-only profile bind

   mount.

4. No service publishes a host port (REQ-DL-13). It depends on the platform `core` profile for

   Nginx and Keycloak only.

Start with `cd compose && docker compose --env-file ../.env up -d`.

### Q.4 Nginx

The platform server block includes `nginx/data-lab.conf`, and the platform Nginx container joins

the `data-lab` network so the upstream names resolve.

| Route | Goes to | Notes |

|---|---|---|

| `/data-lab` | 301 to `/data-lab/` | |

| `/data-lab/` | `data-lab-admin-ui` | |

| `/data-lab/api/` | `data-lab-api` | Forwards `Authorization` |

| `/data-lab/hub/` | `piveau-hub-repo` | Forwards `Authorization` and `X-API-Key` (REQ-DL-11); 16 MB request bodies |

| `/data-lab/search/` | `piveau-hub-search` | |

| `/data-lab/admin`, `/data-lab/admin/` | 301 to `/data-lab/#/admin` | Old bookmarks |

Each route strips its prefix. There is no SPARQL route (REQ-DL-06).

### Q.5 Keycloak Realm

Import `keycloak/realm-dataspace.json` into the platform Keycloak, version 26.4 or newer, then:

1. **`piveau-hub-repo`** (confidential): regenerate its secret into `KEYCLOAK_HUB_CLIENT_SECRET`.

   The import gives it authorization services, its own `uma_protection` role and a service account

   with the `realm-management` roles `manage-users`, `view-users`, `query-users`, `query-groups`,

   `manage-authorization`, `manage-clients`, `view-clients`, `query-clients`, `view-realm` and

   `view-authorization`.

2. **`data-lab-admin`** (public, PKCE `S256`): replace `REPLACE-ME` in its redirect URI and web

   origin with the platform host, keeping the redirect scoped to `/data-lab/*`. **Note:** a wildcard

   redirect on a public client is an open redirect.

3. **`data-lab-api`** (confidential): regenerate its secret into `KEYCLOAK_API_CLIENT_SECRET`. Its

   service account needs `manage-users`, `query-users`, `query-groups` and `view-users`; without

   them every user lookup and membership call answers `502`.

4. **Stewards:** after the hub has first started and created *Piveau System › role: Operator*, an

   administrator adds stewards to it in the admin console (F2).

**Note:** with a role missing from step 1, or Keycloak older than 26.4, the hub logs

`Failed to create Keycloak groups, resource or UMA policies` at start and catalogues get no groups.

The realm export also sets: self-registration off, login with e-mail, password reset on,

brute-force protection on, access tokens of 300 seconds, SSO sessions of 30 minutes idle and

10 hours at most, SSL required for external requests.

### Q.6 Kubernetes

Why: [ADR-0069](/decisions/0069-data-lab-helm-chart-alternative-to-compose.md). The subchart is in the helm

repository from tag `datalab-0.1.0`.

| Topic | Helm deployment |

|---|---|

| Turning it on | `datalab.enabled` and `keycloak.realms.dataspace.enabled`, both `false` by default; set both. With `global.auth.enabled=false` the Data Lab is not deployed |

| Exposure | The UI only, on NodePort `30083` (`global.nodePorts.datalab`), at `/`; it forwards `/api/` to the API. Hub, search, Virtuoso and Elasticsearch are cluster-internal. No `/data-lab/*` route, so REQ-DL-13 applies to Compose only (T15) |

| Stores | One Virtuoso, which also holds the metrics and history graphs; no shadow store. Single-node Elasticsearch with a 512 MB heap. 2 Gi volumes |

| Identity | Realm `dataspace` from the Keycloak chart, mirroring the realm export; client secrets generated into `keycloak-credentials`. The API reaches Keycloak through `KEYCLOAK_INTERNAL_URL` |

| Secrets | Hub and search API keys and the Virtuoso password generated into Secret `datalab-credentials`, kept on `helm uninstall` like the volumes |

| Catalogues | A Job creates `oop-org` and `oop-org-staging` on every install and upgrade, and re-indexes them if they are missing from search |

| Base URI | `datalab.baseUri`; empty means `<publicUrl>/data-lab/`, and `publicUrl` defaults to the NodePort address. Suits kind only; a real deployment sets it before the first install and never changes it (ADR-0069; T16) |

| Images | API and UI `labs.etsi.org:5050/oop/code/data-lab/…:main` with `IfNotPresent`; a real deployment pins `ci-<branch>-<sha>` (ADR-0069; T16) |

| Test users | On kind: `steward` (a steward), `alice`, `bob` |

**Note:** catalogue permissions live in two places, Virtuoso and the realm's groups. To reset the

Data Lab, delete the Virtuoso and Elasticsearch volumes, Secret `datalab-credentials` and realm

`dataspace` together; deleting only one side leaves memberships of catalogues that no longer exist.

### Q.7 Local Trial

`./local/local-up.sh` starts the whole stack with a throwaway Keycloak 26.4 and test users

`steward`, `alice` and `bob` (password `test`). It needs Docker with Compose v2, `curl` and about

4 GB of free memory. Open `http://localhost:11080`. On Linux, add `127.0.0.1 host.docker.internal`

to `/etc/hosts`. `./local/local-down.sh --volumes` also deletes the data.

The development overlay publishes host ports (hub 11091, search 11094, Virtuoso 11890,

Elasticsearch 11920, gateway 11080, API 11081, UI 11082) and adds a gateway that stands in for the

platform Nginx. **Note:** never use the overlays or the local realm in a deployment; their secrets

and users are public.

### Q.8 Operations

| Task | How |

|---|---|

| Create the catalogues | `HUB_URL=… ./bootstrap/bootstrap.sh`. Idempotent; a re-run only refreshes titles and descriptions |

| Verify a deployment | `HUB_URL=… SEARCH_URL=… ./scripts/smoke-test.sh` (§S) |

| Back up | Proposal (F5), not agreed: a nightly dump of both Virtuoso instances to the platform backup target, `isql-v 1111 dba "$VIRTUOSO_DBA_PASSWORD" exec="dump_nquads('dumps', 1, 10000000, 1);"`. Elasticsearch is not backed up |

| Restore | Restore Virtuoso and the matching Keycloak state, then re-index. Procedure and retention remain open (F5); without re-indexing, search serves stale data |

| Re-index | `./scripts/reindex.sh` after a restore, a profile change, or drift between the triple store and the index. It is asynchronous: watch the hub log and compare counts before calling it done. It covers only the two bootstrap catalogues (T13) |

| Profile or shape update | §V.6 |

| Upgrade an image | §V.4 |

| Reset after an AC2 failure | Documented for Helm only (§Q.6); unspecified for Compose |

---

## R. Observability

### R.1 Logging

piveau services are Java/Vert.x and do not use `oop-commons`. They log at `INFO` in a deployment;

the 6G-DALI baseline's `TRACE` level is for development only. The Data Lab API and the UI's Nginx

log in their default formats; the browser app has no server-side logger. Compose keeps two 10 MB log files

per container.

Planned, not configured: JSON log output, and a request id carried from the Nginx access log

through the hub, so a registration can be traced end to end.

### R.2 Metrics

Planned, not built: the Data Lab emits none of these yet.

| Metric | Type | Meaning |

|---|---|---|

| `datalab_registrations_total` | Counter | Registration API writes, by catalogue and outcome (created, updated, rejected) |

| `datalab_validation_failures_total` | Counter | Validation rejections, by shape |

| `datalab_search_requests_total`, `datalab_search_latency_seconds` | Counter, histogram | Search traffic and latency |

| `datalab_private_denied_total` | Counter | Reads of private datasets refused to a caller without rights; logged by the API, not by piveau |

| `datalab_datasets`, `datalab_distributions` | Gauge | Catalogue size, from the triple store |

| `datalab_index_lag_seconds` | Gauge | Difference between triple-store and index counts; a re-index trigger |

Until native Prometheus exporters exist (platform Phase 2 observability), the signals come from the

Nginx access logs and the hub's own metrics.

### R.3 Health Checks

| Service | Target liveness | Target readiness | Probe today |

|---|---|---|---|

| `piveau-hub-repo` | HTTP 200 on `/` | Virtuoso and hub-search reachable | HTTP 200 on `/` (T3) |

| `piveau-hub-search` | HTTP 200 on `/` | Elasticsearch `yellow` or better | HTTP 200 on `/` (T3) |

| Virtuoso, shadow | SPARQL `ASK {}` | Same | HTTP `GET /sparql` without a query |

| Elasticsearch | `/_cluster/health` | Same | `/_cluster/health` answering, whatever its status |

| `data-lab-api` | `/health` | Same | `/health`; it checks no dependency |

| `data-lab-admin-ui` | HTTP 200 on `/` | Static, same | HTTP 200 on `/` |

---

## S. Testing

| Kind | Covers | Runs With |

|---|---|---|

| API unit | Routes and guards, permissions and levels, access levels and their mapping, hidden reads as `404`, promotion and its rollback, membership, public and signed-in search, MAP validation and Turtle round trip, the local realm staying in step with the real one | `make test-api` (pytest; Keycloak and piveau mocked) |

| UI unit | Router, search parameters, form mapping, API error handling | `make test-ui` (TypeScript check, vitest, build) |

| Deployment files | All Compose combinations are valid; every script parses | `make lint-deploy` |

| End to end, local | Stewards and members, public and private catalogues, all three levels, access rights, promotion, and that the hub and search hide private data when called directly | `make local-up`, then `make e2e` |

| Smoke, deployed | Register a public and a private dataset; anonymous read and search see the first and not the second; clean up | `scripts/smoke-test.sh`, after every deployment and image bump |

`make test` runs both unit suites; CI runs the unit suites and the deployment-file checks. Not

covered by any test: the quality targets ([requirements-and-design.md §H.2](/data-lab/requirements-and-design.md)), the Helm chart, client-credentials

scripts, Nginx routing, the dataset page's content, URI minting, and re-indexing

([requirements-and-design.md §H.1](/data-lab/requirements-and-design.md)).

---

## T. Known Gaps

Status is for release 0.1.0. **Open** is an unresolved defect or verification item; **Deferred**

is planned work; **Constraint** is a deliberate release limit. No individual owner or fix release

is assigned; open items remain with the Data Lab maintainers. Publication does not close them.

| Gap | Release status / reason | Workaround |

|---|---|---|

| T1. Direct Registration API writes are not validated against the MAP (REQ-DL-03) | Deferred: Profile mode broke indexing when tested; search was judged more important than validation for this release | Contributors and scripts are responsible for conformance; use the console for checked writes |

| T2. Direct writes carry whatever access level the caller passes (REQ-DL-21) | Open: Recorded as a first-version limit; no reason given | A script writing into a private catalogue must pass `internal` |

| T3. Readiness does not check dependencies; the Virtuoso and Elasticsearch probes are weaker than §R.3's target (REQ-DL-15) | Deferred | None recorded |

| T4. No metrics (§R.2) | Deferred: Waiting for platform Phase 2 observability | Nginx access logs and hub logs |

| T5. No JSON logs and no request id across Nginx and the hub | Deferred | None recorded |

| T6. Controlled vocabularies are fixed in the Data Lab API | Deferred | A code change (§V.1) |

| T7. No audit trail beyond Nginx and application logs | Deferred | None recorded |

| T8. Three fixed levels, although piveau's role configuration could define others | Constraint | — |

| T9. A catalogue's visibility cannot change after creation (REQ-DL-25) | Constraint: By design: changing it would rewrite every dataset's level | None recorded |

| T10. No users from other realms | Constraint | Accounts in realm `dataspace` (E6) |

| T11. AC2 cannot be switched on over existing data | Constraint: A piveau limitation | Enable AC2 from the first start (REQ-DL-26) |

| T12. If the hub answers a catalogue read with an error, the API treats the catalogue as public and can write a private catalogue's dataset as `public` (REQ-DL-21) | Open | None recorded |

| T13. `scripts/reindex.sh` re-indexes only the bootstrap catalogues; which re-index call is supported is unverified (REQ-DL-16) | Open | Unverified: the script uses `PUT /catalogues/{id}/index`; the Helm Job uses `POST /action` with method `index` |

| T14. The API lists at most 100 catalogues; any beyond vanish from lists, search and dataset pages | Open | None recorded |

| T15. The Helm deployment has no platform Nginx: no `/data-lab/*` route, and the hub and search APIs are unreachable from outside the cluster (ADR-0069) | Constraint: ADR-0069: the Data Lab is exposed like the other OOP components | `kubectl port-forward` for scripts. How to run the smoke test against this deployment is unspecified |

| T16. Helm defaults: `baseUri` embeds the NodePort address, which is not served at `/data-lab/`; the Data Lab images use the moving tag `main` | Open | Set `datalab.baseUri` before the first install; pin a tested immutable image tag, as in the deployment guide |

| T17. No high availability for Virtuoso or Elasticsearch | Deferred: Consortium scale; a catalogue outage does not affect OOP service exposure ([requirements-and-design.md §H.2](/data-lab/requirements-and-design.md)) | Restart on failure; back up (F5) |

| T18. An edit through the API loses the stable URIs of distributions and quality measurements (F3) | Open | None recorded |

| T19. Elasticsearch runs without authentication; on Kubernetes no NetworkPolicy isolates the stores | Constraint: Decided for this release: network isolation is the protection ([requirements-and-design.md §J.6](/data-lab/requirements-and-design.md)); NetworkPolicies are deferred by ADR-0069 | Compose: the `data-lab` network. Kubernetes: none recorded |

| T20. Concurrent edits are last-write-wins; racing creates can both succeed | Open | None recorded |

| T21. The dataset page shows no provenance or checksum, and quality timestamps are not displayed (REQ-DL-08) | Deferred | None recorded |

| T22. A direct hub read answers `403` for a hidden dataset and `500` for a missing one, so a caller can tell them apart; raw catalogue counts are not filtered by access level (REQ-DL-21) | Open | Use the Data Lab API or Search API for filtered counts |

| T23. The data-protection rules of [requirements-and-design.md §J.7](/data-lab/requirements-and-design.md) are not checked by the code; the form defaults *Access rights* to Public | Deferred | None recorded |

| T24. An edit through the console drops fields the form lacks, including provenance and checksums ([requirements-and-design.md §K.2](/data-lab/requirements-and-design.md)) | Open | None recorded |

| T25. `GET /users` answers any signed-in user, not only stewards and publishers ([requirements-and-design.md §J.5](/data-lab/requirements-and-design.md)) | Open | None recorded |

| T26. When a promotion and its rollback both fail, the `500` answer does not include the record ([requirements-and-design.md §K.7](/data-lab/requirements-and-design.md)) | Open | None recorded |

| T27. Console edits lose distribution description, licence and measurement technique, including supported input fields | Open | Retain the original record before editing |

| T28. A stale search count of zero allows deletion of a non-empty catalogue without `force` (REQ-DL-20) | Open | Verify the stored records before deleting a catalogue |

| T29. Promotion does not check the production id and can replace an existing record (REQ-DL-23) | Open | Check the target and retain both records before promotion |

| T30. The API and examples use `http://schema.org/`; the MAP uses `https://schema.org/`, whose measured-variable predicates the reader misses | Open | None recorded |

| T31. The Portal link to the Data Lab is not implemented | Deferred: ADR-0049 chooses a link | Open the Data Lab URL directly |

| T32. Base Compose does not forward optional `KEYCLOAK_INTERNAL_URL`, `DL_PERMISSIONS_TTL` or UI `DATA_LAB_API_URL` settings | Open | Add a deployment override; `.env` alone has no effect |

| T33. Python vocabulary lists differ from the vendored SHACL lists; some enumerations are free strings (REQ-DL-22) | Open: which differences are intentional remains F7 | Use the accepted inputs in interface-contract.md §3 |

---

## U. Later Releases

No item below has a scheduled release. "Agreed" means the specification records the intent;

"Proposal" means it does not yet.

### U.1 Every Write Follows the Access Rule and the MAP

- **Missing.** Direct hub writes are neither validated nor held to the stricter access level (T1,

  T2).

- **Plan.** Agreed: a later version applies both to every write. The mechanism is chosen when this

  item is planned: the `dali-*` shapes in the hub's validation hook, or an external validator, such

  as pySHACL, in front of the Registration API. piveau's own validation with the form profile did

  not work when tested ([requirements-and-design.md §L.1](/data-lab/requirements-and-design.md)).

- **Already in place.** The Data Lab API applies its OOP validation subset and access rule, with T12 and T33 outstanding; the shapes are vendored in

  `config/shapes/`; the profile is mounted.

### U.2 Dependency-Aware Readiness

- **Missing.** Readiness equals liveness (T3).

- **Plan.** Agreed (REQ-DL-15). Design unspecified.

- **Already in place.** Probes on every service (§R.3).

### U.3 Metrics and Structured Logs

- **Missing.** No metrics; no JSON logs or request id (T4, T5).

- **Plan.** Agreed target set (§R.2), with platform Phase 2 observability. Design unspecified.

- **Already in place.** `INFO` logging; Nginx access logs.

### U.4 Vocabulary Management

- **Missing.** Stewards cannot change controlled vocabularies (T6).

- **Plan.** Agreed as planned; design unspecified.

- **Already in place.** One definition served at `GET /vocab` for both the form and validation.

### U.5 DataOps Through the Data Lab API

- **Missing.** The DataOps orchestrator still calls piveau and Keycloak directly.

- **Plan.** Proposal (F6): it calls `POST /catalogues` and the member endpoints and holds no

  credentials of its own.

- **Already in place.** The catalogue and member endpoints.

### U.6 Later

The future shape of the Data Lab. The platform roadmap lists the connector, storage, DataOps and

MLOps registration and the quality service as Phase 4 deliverables

([platform/architecture.md](/platform/architecture.md)); none is scheduled.

| Capability | How it attaches to the catalogue |

|---|---|

| Dataspace connector (EDC, Dataspace Protocol) | Connector assets match distributions through `dali:assetId` and `dali:connectorType`; DSP offers are generated from or registered with piveau records (F4) |

| Object storage (S3-compatible data lake) | `dcat:downloadURL` and `dali:assetId` resolve to bucket objects; upload from the dataset form |

| Automated data-quality service (Great Expectations) | Writes `dqv:QualityMeasurement`s to distributions through the Registration API; drives automated promotion from staging |

| DataOps and MLOps pipelines | Register `dcat:DataService`, derived `dcat:Dataset` (PROV-O) and `mldcat:MLModel` with the vendored shapes |

| Governance | Policies on the connector; the catalogue exposes `dct:accessRights` and `odrl:hasPolicy` |

| Platform integration | Portal aggregation of catalogue data; a TF-SDK Intelligence domain `capability_kind = data_product_query` resolving against this catalogue; GAIA-X self-description signing; bridges to external metadata registries |

| Public SPARQL | Served from a separate store that holds public records only ([requirements-and-design.md §K.8](/data-lab/requirements-and-design.md)) |

---

## V. Common Changes

### V.1 Add or Change a Dataset Form Field or Controlled Value

1. Check the field against the vendored MAP. OOP-specific fields are a `dali:` or `oop:`

   extension, never a fork (E2).

2. In `api/app/mapform.py`, change `CONTEXT_FIELDS` (testbed context) or the vocabulary lists, the

   `DatasetIn` or `DistributionIn` model, `validate` and `build_graph`.

3. If the dataset page must show it, update `parse_dataset` in `api/app/datasets.py`.

4. For a testbed-context field, the UI needs no change: the form renders those fields from

   `/vocab`. For any other field, update `admin-ui/src/types.ts` and `form.ts`.

5. Update [interface-contract.md §3](/data-lab/interface-contract.md), and the field count in §2.6.

- **Files:** `api/app/mapform.py`, `api/app/datasets.py`, `admin-ui/src/`.

- **Check:** [requirements-and-design.md §L](/data-lab/requirements-and-design.md), REQ-DL-22,

  [ADR-0070](/decisions/0070-data-lab-piveau-hub-and-6gdali-map.md).

- **Tests:** `api/tests/test_mapform.py`, `admin-ui/src/form.test.ts`; run `make test`.

- **Elsewhere:** none. **Note:** existing records are not re-validated.

### V.2 Add or Change a Data Lab API Endpoint

1. Add the route in `api/app/main.py` with the right guard from `auth.py`: `require_user`,

   `require_catalogue_access` (contributor), `require_catalogue_manage` (publisher) or

   `require_admin` (steward). A route without a guard is anonymous.

2. Put the logic in the module that owns it (`hub.py`, `datasets.py`, `public.py`,

   `keycloak_admin.py`).

3. Reads of user-visible data forward the caller's token; use the hub API key only for writes the

   route has already authorised and for management reads ([requirements-and-design.md §J.4](/data-lab/requirements-and-design.md)).

4. Update the UI's `api.ts` if the UI calls it.

5. Update [requirements-and-design.md §I.3](/data-lab/requirements-and-design.md) and

   [interface-contract.md §2](/data-lab/interface-contract.md).

- **Tests:** `api/tests/test_api.py` (guards), plus the module's tests; `local/e2e.py` for

  behaviour that piveau enforces. Run `make test-api`; `make local-up && make e2e`.

- **Elsewhere:** none; Nginx forwards everything under `/data-lab/api/`.

### V.3 Change Who May Do What

1. Levels map to role groups in `LEVEL_ROLES` (`api/app/keycloak_admin.py`) and to scopes in

   `levels` and `owned_catalogues` (`api/app/auth.py`). Keep the two in step.

2. Steward rights are `catalog:create` on `0_global` (`auth.is_admin`).

3. **Note:** piveau creates and removes the role groups. The API only changes membership

   (REQ-DL-17); never create or delete the groups from the API.

4. Update [requirements-and-design.md §J](/data-lab/requirements-and-design.md) and REQ-DL-17,

   REQ-DL-18 and REQ-DL-23 if the rule changes.

- **Tests:** `api/tests/test_auth.py`, `test_api.py`, `local/e2e.py`.

- **Elsewhere:** the Helm Keycloak chart and `keycloak/realm-dataspace.json` if client roles

  change; regenerate the local realm with `make realm`.

### V.4 Bump a Pinned Upstream Image

1. Read the upstream changelog. **Note:** a `piveau-hub-repo` minor version has renamed settings

   before; a Virtuoso version that changes the on-disk format needs a dump and restore, not the old

   volume.

2. Change one image at a time: `.env.example`, the default in `compose/docker-compose.yml`, and the

   Helm `values.yaml` of `charts/datalab`.

3. Run `scripts/smoke-test.sh` after each, and `make e2e` locally.

4. Update §Q.1 with the new pin.

- **Note:** never `:latest`. AC2 must stay on; it cannot be switched on over existing data.

- **Elsewhere:** the helm repository.

### V.5 Add a Deployment Setting

1. Read it in `api/app/config.py`, or add it to the UI's `docker-entrypoint.d/40-admin-config.sh`

   and `src/config.ts` (and to `NGINX_ENVSUBST_FILTER` if the Nginx template uses it).

2. Pass it in `compose/docker-compose.yml` and document it in `.env.example`.

3. Set it in the Helm templates and `values.yaml` of `charts/datalab`.

4. Document it in `docs/deployment.md` and in §P.

- **Tests:** `make lint-deploy`; unit tests where the setting changes behaviour.

- **Elsewhere:** the helm repository; the platform secret store for secrets.

### V.6 Take a New MAP Version or Shape Update

1. Re-vendor the profile and shapes into the architecture repository's `external-specs/`, following

   [ADR-0026](/decisions/0026-external-specs-vendoring-with-commit-pinning.md) and the re-vendor

   checklist in [external-specs/index.md](/external-specs/index.md), which is written for CAMARA

   snapshots and applies here where it fits.

2. Copy the shapes into `config/shapes/` in the data-lab repository. If the piveau form and search

   profile in `config/profile/` must follow, update it too; it is inactive while profile mode is off

   ([requirements-and-design.md §L.1](/data-lab/requirements-and-design.md)).

3. If mandatory fields or vocabularies changed, update the API's rules (§V.1).

4. Re-index (§Q.8). Records registered under an old rule are not re-validated automatically, and

   no tool re-validates stored records; how to do it is unspecified.

- **Check:** E2 in [overview-and-architecture.md §E](/data-lab/overview-and-architecture.md), [requirements-and-design.md §L](/data-lab/requirements-and-design.md).

- **Tests:** `api/tests/test_mapform.py`; `make e2e`.

---

## Revision History

| Version | Date | Summary |

|---------|------|---------|

| 0.1.0 | 2026-10-16 | First release. |

# Data Lab (DL)

Released catalogue specification (`version: "0.1.0"`, [ADR-0071](/decisions/0071-data-lab-own-version-outside-release-pin.md)).

Four-part spec; read in order. Requirement status is in Requirements & Design §H.1;

open questions in Overview §F; implementation gaps in Implementation & Delivery §T.

* [Overview & Architecture](overview-and-architecture.md) - what the Data Lab is, scope and ownership, who sees what, terms, features, boundaries, assumptions, open questions

* [Requirements & Design](requirements-and-design.md) - requirements, components and interfaces, access control, flows, metadata model, lifecycle, errors

* [Implementation & Delivery](implementation-and-delivery.md) - code layout, configuration, deployment, observability, testing, known gaps, later releases, common changes

* [Interface Contract](interface-contract.md) - Data Lab API details, the dataset record, piveau API usage, worked examples

Normative metadata reference: [6G-DALI Metadata Application Profile (vendored snapshot)](/external-specs/6gdali-metadata-application-profile/VERSION.md) - DCAT-AP 3.0 + `dali:` testbed context + PROV-O + DQV, with layered SHACL shapes and the `dali:` ontology (ADR-0026, ADR-0070)