From 4c01c4b68e75b4637a63f96dd6b4008799a294d5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Guillermo=20Sanz=20L=C3=B3pez?= Date: Thu, 12 Feb 2026 13:01:57 +0100 Subject: [PATCH 1/3] certs generation release notes --- doc/releasenotes.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/doc/releasenotes.md b/doc/releasenotes.md index cfe9e5ea..4abdea0c 100644 --- a/doc/releasenotes.md +++ b/doc/releasenotes.md @@ -7,6 +7,18 @@ ### **Technical Debt Solved** +#### **Certificate Generation Reworked for Multi-Instance Deployments (Common Vault)** + +The certificate generation and storage flow has been updated to support **multiple CAPIF instances sharing the same Vault**. +Previously, the startup and certificate provisioning logic assumed a single CAPIF instance and stored certificates from fixed Vault paths, which caused collisions when deploying more than one instance using a common Vault. + +With this change: +- Certificates and keys are now **generated at the service level** (e.g., NGINX generates it's own key and CSR locally). +- Vault is now used only as a **signing authority (CA)** to sign incoming CSRs and to store the resulting artifacts, avoiding Vault-specific instance coupling and enabling the same Vault to serve other CAPIF deployments. +- A **unique CCF identifier (ccf_id)** is used as the namespace key to store and retrieve CAPIF certificates. +- CAPIF-related certificates are stored under instance-scoped Vault paths (e.g. `secret/capif//...`) to prevent overwriting assets across deployments. +- Startup scripts and tooling were updated to obtain and use the correct `ccf_id` dynamically, ensuring each instance loads the correct certificate material. + #### **NGINX Configuration improved** The NGINX configuration included in the OpenCAPIF deployment has been improved. -- GitLab From 784975726e3dea9792a69804f3db5470cb1992a8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Guillermo=20Sanz=20L=C3=B3pez?= Date: Wed, 18 Feb 2026 13:52:31 +0100 Subject: [PATCH 2/3] vault certificate mngmt --- doc/vault/vault.md | 296 +++++++++++++++++++++++++++++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 297 insertions(+) create mode 100644 doc/vault/vault.md diff --git a/doc/vault/vault.md b/doc/vault/vault.md new file mode 100644 index 00000000..a62dd228 --- /dev/null +++ b/doc/vault/vault.md @@ -0,0 +1,296 @@ +# CAPIF Certificate Generation Architecture + +## Summary + +This document describes the changes made to the certificate generation and management architecture in CAPIF, implemented in the `OCF182-certs-generation` branch. The main objective is to allow a single Vault server to serve multiple CAPIF instances efficiently and securely. + +## Main Changes + +### Previous Architecture + +In the previous implementation, Vault was responsible for: +- Generating the root CA and intermediate CA +- **Generating** service certificates +- Storing and distributing certificates + +**Problems:** +- Vault generated service certificates (less secure) +- Difficult scalability for multiple CAPIF instances +- Strong coupling between Vault and each CAPIF instance + +### New Architecture + +In the new implementation: + +``` +┌──────────────────────────────────────────────────────────────┐ +│ VAULT │ +│ ┌────────────────────────────────────────────────────────┐ │ +│ │ 1. Generates Root CA │ │ +│ │ 2. Generates Intermediate CA │ │ +│ │ 3. Configures signing role │ │ +│ │ 4. Stores CA bundle in secret/ca │ │ +│ └────────────────────────────────────────────────────────┘ │ +│ │ +│ Functions: │ +│ ✓ SIGNS certificates (receives CSR) │ +│ ✓ STORES signed certificates │ +└──────────────────────────────────────────────────────────────┘ + ▲ + │ + ┌────────────────────┼────────────────────┐ + │ │ │ + ▼ ▼ ▼ +┌──────────────┐ ┌──────────────┐ ┌──────────────┐ +│ CAPIF #1 │ │ CAPIF #2 │ │ CAPIF #N │ +│ CCF_ID: A │ │ CCF_ID: B │ │ CCF_ID: C │ +└──────────────┘ └──────────────┘ └──────────────┘ +``` + +**Benefits:** +- ✅ Enhanced security: each service generates its own private key +- ✅ Scalability: one Vault serves multiple CAPIF instances +- ✅ Isolation: certificates organized by CCF_ID +- ✅ PKI best practices: separation between generation and signing + +## Modified Components + +### 1. Vault (`services/vault/vault_prepare_certs.sh`) + +**Main changes:** +- No longer **generates** service certificates, only PKI infrastructure +- Generates only root CA and intermediate CA +- Configures PKI signing endpoint at `/v1/pki_int/sign/my-ca` +- Stores CA bundle in `secret/ca` (KV v2) +- Configures `my-ca` signing role with flexible policies that allow signing any CSR + +### 2. NGINX (`services/nginx/nginx_prepare.sh`) + +**New primary certificate generation component:** + +Implemented flow: +1. Fetches CA certificate from Vault (`secret/ca`) +2. Generates its own private key (`server.key`) if it doesn't exist +3. Creates a CSR (Certificate Signing Request) with CAPIF hostname +4. Sends CSR to Vault for signing +5. Receives signed certificate (`server.crt`) +6. Extracts public key from certificate (`server_pub.pem`) +7. Obtains unique CCF_ID from Helper +8. Stores all certificates in Vault under `secret/capif/${CCF_ID}/nginx` + +**Important note:** This script includes retries with timeout to handle ordered service startups. + +### 3. Register (`services/register/register_prepare.sh`) + +**Similar pattern to NGINX:** +- Generates its own private key (`register_key.key`) +- Creates CSR with complete organization information +- Obtains CA from Vault +- Requests certificate signing from Vault via PKI endpoint +- Saves signed certificate locally + +### 4. Helper Service (`services/helper/helper_service/app.py`) + +**Modifications at application startup:** +- Creates a dedicated `certs/` directory with restrictive permissions +- Generates key pair and CSR for superadmin certificate using pyOpenSSL +- Saves private key locally with 600 permissions +- Requests certificate signing from Vault +- Downloads and stores signed superadmin certificate +- Downloads and saves CA root from Vault + +**Security improvements:** permission validation and robust error handling. + +### 5. API Services (Invoker, Provider, Security) + +**Modified `prepare_*.sh` scripts:** + +New consumption pattern (no longer generate certificates): +1. Query Helper to obtain CAPIF instance's CCF_ID +2. Retrieve server's public key from Vault using path `secret/capif/${CCF_ID}/nginx` +3. Save public key to local location +4. Implement robust retries to wait for NGINX to generate and store certificates + +**Affected files:** +- `prepare_invoker.sh` +- `prepare_provider.sh` +- `prepare_security.sh` + +## Certificate Organization in Vault + +### KV (Key-Value v2) Structure + +**Root level:** +- `secret/ca`: Stores intermediate CA certificate (accessible by all CAPIF instances) + +**Per-CAPIF level:** +- `secret/capif//nginx/`: Contains certificates for each CAPIF instance + - `server_crt`: Server certificate + - `server_key`: Server private key + - `server_pub`: Extracted public key + - `ca`: Copy of CA bundle + +**Note:** In the future, sub-levels for `register/` and `helper/` will be added under each CCF_ID. + +### PKI Endpoints + +**Root PKI Engine (`pki/`):** +- `root/generate/internal`: Generates root CA +- `config/urls`: Configures issuance and CRL URLs +- `root/sign-intermediate`: Signs intermediate CA + +**Intermediate PKI Engine (`pki_int/`):** +- `intermediate/generate/internal`: Generates intermediate CA CSR +- `intermediate/set-signed`: Installs signed intermediate certificate +- `roles/my-ca`: Defines signing role with its policies +- `sign/my-ca`: Endpoint used by services to sign their CSRs + +## Deployment Flow + +### 1. Vault Initialization + +**Execute once per Vault cluster:** +- In Docker: run `vault_prepare_certs.sh` script +- In Kubernetes: apply ConfigMap and Job from `helm/vault-job/vault-job.yaml` + +**Expected result:** +- ✅ Root CA generated +- ✅ Intermediate CA generated and signed +- ✅ `my-ca` role configured +- ✅ CA bundle stored in `secret/ca` + +### 2. CAPIF Instance Deployment + +Each CAPIF instance (identified by unique CCF_ID): + +1. **Helper starts** → Generates unique CCF_ID +2. **NGINX starts:** + - Generates `server.key` and `server.csr` + - Requests signing from Vault + - Receives `server.crt` + - Stores in `secret/capif/${CCF_ID}/nginx` +3. **Register starts:** + - Generates registration certificate + - Requests signing from Vault +4. **API Services start:** + - Retrieve certificates from Vault using CCF_ID + - Use public key for validation + +## Advantages of New Architecture + +### Security + +1. **Private keys never transit**: each service generates its private key locally +2. **Principle of least privilege**: Vault only has CA role, doesn't manage service keys +3. **Isolation per CAPIF**: certificates separated by CCF_ID + +### Scalability + +1. **One Vault, multiple CAPIF**: total decoupling +2. **Namespace per instance**: `secret/capif/${CCF_ID}/` +3. **No collisions**: unique CCF_ID guarantees separation + +### Operations + +1. **Simplified rotation**: each CAPIF can regenerate its certificates independently +2. **Improved auditing**: can track which CAPIF requested which certificate +3. **Independent deployment**: a CAPIF can restart without affecting others + +## Helm/Kubernetes Compatibility + +The `helm/vault-job/vault-job.yaml` file has been significantly simplified: + +**Main changes:** +- ❌ No longer generates certificates for specific services +- ✅ Only executes basic PKI setup (Root CA, Intermediate CA, signing role) +- ✅ Lighter: 144 lines removed +- ✅ ConfigMap contains a self-contained script that enables PKI engines, generates CAs and configures signing role +- ✅ Job runs only once when deploying Vault namespace + +## Docker Compose Configuration + +**Changes in `services/docker-compose-capif.yml`:** + +**NGINX:** +- No longer mounts external certificate volumes +- Certificates are generated dynamically in `/etc/nginx/certs` at startup +- New environment variables: `VAULT_HOSTNAME`, `VAULT_PORT`, `VAULT_ACCESS_TOKEN`, `CAPIF_HOSTNAME` + +**Helper:** +- Manages its own certificate directory internally +- `certs/` directory is automatically created by `app.py` at startup +- No longer requires external volumes for certificates + +## Migration from Previous Architecture + +### Migration steps: + +1. **Backup existing certificates**: Export current certificates from Vault (optional, for history) +2. **Deploy new Vault**: Execute updated `vault_prepare_certs.sh` script +3. **Clean old certificates**: Remove certificate directories in NGINX and Register services +4. **Redeploy CAPIF services**: Execute `run.sh` so services automatically generate their new certificates + +### Important considerations: + +- ⚠️ **Downtime**: There will be a service interruption during migration +- ⚠️ **New CCF_ID**: Each instance will get a new unique CCF_ID +- ⚠️ **Regenerated certificates**: All service certificates will be new +- ✅ **No API changes**: CAPIF's public interface remains the same +- ✅ **Compatibility**: Existing clients will continue working after obtaining new CA + +## Testing + +**Recommended verifications:** + +1. **Verify CA in Vault**: Query `secret/data/ca` with curl to confirm CA bundle is available +2. **Start CAPIF**: Execute startup script and observe logs from each service +3. **Verify NGINX certificate**: Use `openssl x509` inside container to inspect generated certificate +4. **Verify storage in Vault**: Obtain CCF_ID from Helper and query path `secret/capif/${CCF_ID}/nginx` in Vault +5. **Run test suite**: Run `run_capif_tests.sh` to validate complete functionality + +## Troubleshooting + +### Error: "Unable to retrieve CA certificate from Vault" + +**Cause:** Vault has not been properly initialized with PKI infrastructure + +**Solution:** Execute Vault preparation script (`vault_prepare_certs.sh`) inside container + +### Error: "Failed to sign server certificate" + +**Cause:** `my-ca` signing role is not configured or Vault token doesn't have sufficient permissions + +**Solution:** Verify role exists and token has write permissions on `pki_int/sign/my-ca` + +### Error: "Unable to retrieve CCF_ID from Helper" + +**Cause:** Helper service hasn't completed initialization yet + +**Solution:** Wait 30-60 seconds. Service scripts implement automatic retries with incremental delays + +## References + +### Modified Files (Branch OCF182-certs-generation) + +- [`services/vault/vault_prepare_certs.sh`](../services/vault/vault_prepare_certs.sh) - PKI Setup +- [`services/nginx/nginx_prepare.sh`](../services/nginx/nginx_prepare.sh) - NGINX certs generation +- [`services/register/register_prepare.sh`](../services/register/register_prepare.sh) - Register certs generation +- [`services/helper/helper_service/app.py`](../services/helper/helper_service/app.py) - Superadmin cert generation +- [`services/TS29222_CAPIF_API_Invoker_Management_API/prepare_invoker.sh`](../services/TS29222_CAPIF_API_Invoker_Management_API/prepare_invoker.sh) - Cert consumption +- [`services/TS29222_CAPIF_API_Provider_Management_API/prepare_provider.sh`](../services/TS29222_CAPIF_API_Provider_Management_API/prepare_provider.sh) - Cert consumption +- [`services/TS29222_CAPIF_Security_API/prepare_security.sh`](../services/TS29222_CAPIF_Security_API/prepare_security.sh) - Cert consumption +- [`helm/vault-job/vault-job.yaml`](../helm/vault-job/vault-job.yaml) - Kubernetes Job +- [`services/docker-compose-capif.yml`](../services/docker-compose-capif.yml) - Docker Configuration + +### Related Documentation + +- [HashiCorp Vault PKI Secrets Engine](https://developer.hashicorp.com/vault/docs/secrets/pki) +- [OpenSSL CSR Generation](https://www.openssl.org/docs/man1.1.1/man1/openssl-req.html) +- [CAPIF Specification TS 29.222](https://www.3gpp.org/DynaReport/29222.htm) + +--- + +**Document Version**: 1.0 +**Date**: February 2026 +**Branch**: OCF182-certs-generation +**Author**: CAPIF Team diff --git a/mkdocs.yml b/mkdocs.yml index aaf20b7a..44fd63df 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -98,6 +98,7 @@ nav: - Event Filter: ./event-filter/event-filter.md - Dynamic Configuration: ./configuration/configuration.md - Event Reporting Information: ./event-req/event-req.md + - Vault Certificate Management: ./vault/vault.md - SDK: - Introduction: ./sdk/sdk_introduction.md - Requirements: ./sdk/sdk_requirements.md -- GitLab From 3cdc90c7af0799986f52abc7edf9b3e5cf65fd67 Mon Sep 17 00:00:00 2001 From: Jorge Moratinos Salcines Date: Wed, 18 Feb 2026 17:01:35 +0100 Subject: [PATCH 3/3] minor fix due to renderer issues --- doc/releasenotes.md | 1 + doc/vault/vault.md | 39 ++++++++++++++++++++++++++++++--------- 2 files changed, 31 insertions(+), 9 deletions(-) diff --git a/doc/releasenotes.md b/doc/releasenotes.md index 4abdea0c..8aad1ad7 100644 --- a/doc/releasenotes.md +++ b/doc/releasenotes.md @@ -13,6 +13,7 @@ The certificate generation and storage flow has been updated to support **multip Previously, the startup and certificate provisioning logic assumed a single CAPIF instance and stored certificates from fixed Vault paths, which caused collisions when deploying more than one instance using a common Vault. With this change: + - Certificates and keys are now **generated at the service level** (e.g., NGINX generates it's own key and CSR locally). - Vault is now used only as a **signing authority (CA)** to sign incoming CSRs and to store the resulting artifacts, avoiding Vault-specific instance coupling and enabling the same Vault to serve other CAPIF deployments. - A **unique CCF identifier (ccf_id)** is used as the namespace key to store and retrieve CAPIF certificates. diff --git a/doc/vault/vault.md b/doc/vault/vault.md index a62dd228..7b2db522 100644 --- a/doc/vault/vault.md +++ b/doc/vault/vault.md @@ -9,11 +9,13 @@ This document describes the changes made to the certificate generation and manag ### Previous Architecture In the previous implementation, Vault was responsible for: + - Generating the root CA and intermediate CA - **Generating** service certificates - Storing and distributing certificates **Problems:** + - Vault generated service certificates (less secure) - Difficult scalability for multiple CAPIF instances - Strong coupling between Vault and each CAPIF instance @@ -48,6 +50,7 @@ In the new implementation: ``` **Benefits:** + - ✅ Enhanced security: each service generates its own private key - ✅ Scalability: one Vault serves multiple CAPIF instances - ✅ Isolation: certificates organized by CCF_ID @@ -58,6 +61,7 @@ In the new implementation: ### 1. Vault (`services/vault/vault_prepare_certs.sh`) **Main changes:** + - No longer **generates** service certificates, only PKI infrastructure - Generates only root CA and intermediate CA - Configures PKI signing endpoint at `/v1/pki_int/sign/my-ca` @@ -69,6 +73,7 @@ In the new implementation: **New primary certificate generation component:** Implemented flow: + 1. Fetches CA certificate from Vault (`secret/ca`) 2. Generates its own private key (`server.key`) if it doesn't exist 3. Creates a CSR (Certificate Signing Request) with CAPIF hostname @@ -83,6 +88,7 @@ Implemented flow: ### 3. Register (`services/register/register_prepare.sh`) **Similar pattern to NGINX:** + - Generates its own private key (`register_key.key`) - Creates CSR with complete organization information - Obtains CA from Vault @@ -92,6 +98,7 @@ Implemented flow: ### 4. Helper Service (`services/helper/helper_service/app.py`) **Modifications at application startup:** + - Creates a dedicated `certs/` directory with restrictive permissions - Generates key pair and CSR for superadmin certificate using pyOpenSSL - Saves private key locally with 600 permissions @@ -106,12 +113,14 @@ Implemented flow: **Modified `prepare_*.sh` scripts:** New consumption pattern (no longer generate certificates): + 1. Query Helper to obtain CAPIF instance's CCF_ID 2. Retrieve server's public key from Vault using path `secret/capif/${CCF_ID}/nginx` 3. Save public key to local location 4. Implement robust retries to wait for NGINX to generate and store certificates **Affected files:** + - `prepare_invoker.sh` - `prepare_provider.sh` - `prepare_security.sh` @@ -121,9 +130,11 @@ New consumption pattern (no longer generate certificates): ### KV (Key-Value v2) Structure **Root level:** + - `secret/ca`: Stores intermediate CA certificate (accessible by all CAPIF instances) **Per-CAPIF level:** + - `secret/capif//nginx/`: Contains certificates for each CAPIF instance - `server_crt`: Server certificate - `server_key`: Server private key @@ -135,11 +146,13 @@ New consumption pattern (no longer generate certificates): ### PKI Endpoints **Root PKI Engine (`pki/`):** + - `root/generate/internal`: Generates root CA - `config/urls`: Configures issuance and CRL URLs - `root/sign-intermediate`: Signs intermediate CA **Intermediate PKI Engine (`pki_int/`):** + - `intermediate/generate/internal`: Generates intermediate CA CSR - `intermediate/set-signed`: Installs signed intermediate certificate - `roles/my-ca`: Defines signing role with its policies @@ -150,10 +163,12 @@ New consumption pattern (no longer generate certificates): ### 1. Vault Initialization **Execute once per Vault cluster:** + - In Docker: run `vault_prepare_certs.sh` script - In Kubernetes: apply ConfigMap and Job from `helm/vault-job/vault-job.yaml` **Expected result:** + - ✅ Root CA generated - ✅ Intermediate CA generated and signed - ✅ `my-ca` role configured @@ -165,14 +180,17 @@ Each CAPIF instance (identified by unique CCF_ID): 1. **Helper starts** → Generates unique CCF_ID 2. **NGINX starts:** + - Generates `server.key` and `server.csr` - Requests signing from Vault - Receives `server.crt` - Stores in `secret/capif/${CCF_ID}/nginx` 3. **Register starts:** + - Generates registration certificate - Requests signing from Vault 4. **API Services start:** + - Retrieve certificates from Vault using CCF_ID - Use public key for validation @@ -201,6 +219,7 @@ Each CAPIF instance (identified by unique CCF_ID): The `helm/vault-job/vault-job.yaml` file has been significantly simplified: **Main changes:** + - ❌ No longer generates certificates for specific services - ✅ Only executes basic PKI setup (Root CA, Intermediate CA, signing role) - ✅ Lighter: 144 lines removed @@ -212,11 +231,13 @@ The `helm/vault-job/vault-job.yaml` file has been significantly simplified: **Changes in `services/docker-compose-capif.yml`:** **NGINX:** + - No longer mounts external certificate volumes - Certificates are generated dynamically in `/etc/nginx/certs` at startup - New environment variables: `VAULT_HOSTNAME`, `VAULT_PORT`, `VAULT_ACCESS_TOKEN`, `CAPIF_HOSTNAME` **Helper:** + - Manages its own certificate directory internally - `certs/` directory is automatically created by `app.py` at startup - No longer requires external volumes for certificates @@ -272,15 +293,15 @@ The `helm/vault-job/vault-job.yaml` file has been significantly simplified: ### Modified Files (Branch OCF182-certs-generation) -- [`services/vault/vault_prepare_certs.sh`](../services/vault/vault_prepare_certs.sh) - PKI Setup -- [`services/nginx/nginx_prepare.sh`](../services/nginx/nginx_prepare.sh) - NGINX certs generation -- [`services/register/register_prepare.sh`](../services/register/register_prepare.sh) - Register certs generation -- [`services/helper/helper_service/app.py`](../services/helper/helper_service/app.py) - Superadmin cert generation -- [`services/TS29222_CAPIF_API_Invoker_Management_API/prepare_invoker.sh`](../services/TS29222_CAPIF_API_Invoker_Management_API/prepare_invoker.sh) - Cert consumption -- [`services/TS29222_CAPIF_API_Provider_Management_API/prepare_provider.sh`](../services/TS29222_CAPIF_API_Provider_Management_API/prepare_provider.sh) - Cert consumption -- [`services/TS29222_CAPIF_Security_API/prepare_security.sh`](../services/TS29222_CAPIF_Security_API/prepare_security.sh) - Cert consumption -- [`helm/vault-job/vault-job.yaml`](../helm/vault-job/vault-job.yaml) - Kubernetes Job -- [`services/docker-compose-capif.yml`](../services/docker-compose-capif.yml) - Docker Configuration +- [`services/vault/vault_prepare_certs.sh`](https://labs.etsi.org/rep/ocf/capif/-/blob/staging/services/vault/vault_prepare_certs.sh) - PKI Setup +- [`services/nginx/nginx_prepare.sh`](https://labs.etsi.org/rep/ocf/capif/-/blob/staging/services/nginx/nginx_prepare.sh) - NGINX certs generation +- [`services/register/register_prepare.sh`](https://labs.etsi.org/rep/ocf/capif/-/blob/staging/services/register/register_prepare.sh) - Register certs generation +- [`services/helper/helper_service/app.py`](https://labs.etsi.org/rep/ocf/capif/-/blob/staging/services/helper/helper_service/app.py) - Superadmin cert generation +- [`services/TS29222_CAPIF_API_Invoker_Management_API/prepare_invoker.sh`](https://labs.etsi.org/rep/ocf/capif/-/blob/staging/services/TS29222_CAPIF_API_Invoker_Management_API/prepare_invoker.sh) - Cert consumption +- [`services/TS29222_CAPIF_API_Provider_Management_API/prepare_provider.sh`](https://labs.etsi.org/rep/ocf/capif/-/blob/staging/services/TS29222_CAPIF_API_Provider_Management_API/prepare_provider.sh) - Cert consumption +- [`services/TS29222_CAPIF_Security_API/prepare_security.sh`](https://labs.etsi.org/rep/ocf/capif/-/blob/staging/services/TS29222_CAPIF_Security_API/prepare_security.sh) - Cert consumption +- [`helm/vault-job/vault-job.yaml`](https://labs.etsi.org/rep/ocf/capif/-/blob/staging/helm/vault-job/vault-job.yaml) - Kubernetes Job +- [`services/docker-compose-capif.yml`](https://labs.etsi.org/rep/ocf/capif/-/blob/staging/services/docker-compose-capif.yml) - Docker Configuration ### Related Documentation -- GitLab