Merge branch...

### **Technical Debt Solved**

#### **Certificate Generation Reworked for Multi-Instance Deployments (Common Vault)**

The certificate generation and storage flow has been updated to support **multiple CAPIF instances sharing the same Vault**.

Previously, the startup and certificate provisioning logic assumed a single CAPIF instance and stored certificates from fixed Vault paths, which caused collisions when deploying more than one instance using a common Vault.

With this change:

- Certificates and keys are now **generated at the service level** (e.g., NGINX generates it's own key and CSR locally).

- Vault is now used only as a **signing authority (CA)** to sign incoming CSRs and to store the resulting artifacts, avoiding Vault-specific instance coupling and enabling the same Vault to serve other CAPIF deployments.

- A **unique CCF identifier (ccf_id)** is used as the namespace key to store and retrieve CAPIF certificates.

- CAPIF-related certificates are stored under instance-scoped Vault paths (e.g. `secret/capif/<ccf_id>/...`) to prevent overwriting assets across deployments.

- Startup scripts and tooling were updated to obtain and use the correct `ccf_id` dynamically, ensuring each instance loads the correct certificate material.

#### **NGINX Configuration improved**

The NGINX configuration included in the OpenCAPIF deployment has been improved.

# CAPIF Certificate Generation Architecture

## Summary

This document describes the changes made to the certificate generation and management architecture in CAPIF, implemented in the `OCF182-certs-generation` branch. The main objective is to allow a single Vault server to serve multiple CAPIF instances efficiently and securely.

## Main Changes

### Previous Architecture

In the previous implementation, Vault was responsible for:

- Generating the root CA and intermediate CA

- **Generating** service certificates

- Storing and distributing certificates

**Problems:**

- Vault generated service certificates (less secure)

- Difficult scalability for multiple CAPIF instances

- Strong coupling between Vault and each CAPIF instance

### New Architecture

In the new implementation:

```

┌──────────────────────────────────────────────────────────────┐

│                          VAULT                               │

│  ┌────────────────────────────────────────────────────────┐  │

│  │  1. Generates Root CA                                  │  │

│  │  2. Generates Intermediate CA                          │  │

│  │  3. Configures signing role                            │  │

│  │  4. Stores CA bundle in secret/ca                      │  │

│  └────────────────────────────────────────────────────────┘  │

│                                                              │

│  Functions:                                                  │

│  ✓ SIGNS certificates (receives CSR)                         │

│  ✓ STORES signed certificates                                │

└──────────────────────────────────────────────────────────────┘

                             ▲

                             │

        ┌────────────────────┼────────────────────┐

        │                    │                    │

        ▼                    ▼                    ▼

┌──────────────┐    ┌──────────────┐    ┌──────────────┐

│  CAPIF #1    │    │  CAPIF #2    │    │  CAPIF #N    │

│  CCF_ID: A   │    │  CCF_ID: B   │    │  CCF_ID: C   │

└──────────────┘    └──────────────┘    └──────────────┘

```

**Benefits:**

- ✅ Enhanced security: each service generates its own private key

- ✅ Scalability: one Vault serves multiple CAPIF instances

- ✅ Isolation: certificates organized by CCF_ID

- ✅ PKI best practices: separation between generation and signing

## Modified Components

### 1. Vault (`services/vault/vault_prepare_certs.sh`)

**Main changes:**

- No longer **generates** service certificates, only PKI infrastructure

- Generates only root CA and intermediate CA

- Configures PKI signing endpoint at `/v1/pki_int/sign/my-ca`

- Stores CA bundle in `secret/ca` (KV v2)

- Configures `my-ca` signing role with flexible policies that allow signing any CSR

### 2. NGINX (`services/nginx/nginx_prepare.sh`)

**New primary certificate generation component:**

Implemented flow:

1. Fetches CA certificate from Vault (`secret/ca`)

2. Generates its own private key (`server.key`) if it doesn't exist

3. Creates a CSR (Certificate Signing Request) with CAPIF hostname

4. Sends CSR to Vault for signing

5. Receives signed certificate (`server.crt`)

6. Extracts public key from certificate (`server_pub.pem`)

7. Obtains unique CCF_ID from Helper

8. Stores all certificates in Vault under `secret/capif/${CCF_ID}/nginx`

**Important note:** This script includes retries with timeout to handle ordered service startups.

### 3. Register (`services/register/register_prepare.sh`)

**Similar pattern to NGINX:**

- Generates its own private key (`register_key.key`)

- Creates CSR with complete organization information

- Obtains CA from Vault

- Requests certificate signing from Vault via PKI endpoint

- Saves signed certificate locally

### 4. Helper Service (`services/helper/helper_service/app.py`)

**Modifications at application startup:**

- Creates a dedicated `certs/` directory with restrictive permissions

- Generates key pair and CSR for superadmin certificate using pyOpenSSL

- Saves private key locally with 600 permissions

- Requests certificate signing from Vault

- Downloads and stores signed superadmin certificate

- Downloads and saves CA root from Vault

**Security improvements:** permission validation and robust error handling.

### 5. API Services (Invoker, Provider, Security)

**Modified `prepare_*.sh` scripts:**

New consumption pattern (no longer generate certificates):

1. Query Helper to obtain CAPIF instance's CCF_ID

2. Retrieve server's public key from Vault using path `secret/capif/${CCF_ID}/nginx`

3. Save public key to local location

4. Implement robust retries to wait for NGINX to generate and store certificates

**Affected files:**

- `prepare_invoker.sh`

- `prepare_provider.sh`

- `prepare_security.sh`

## Certificate Organization in Vault

### KV (Key-Value v2) Structure

**Root level:**

- `secret/ca`: Stores intermediate CA certificate (accessible by all CAPIF instances)

**Per-CAPIF level:**

- `secret/capif/<CCF_ID>/nginx/`: Contains certificates for each CAPIF instance

  - `server_crt`: Server certificate

  - `server_key`: Server private key

  - `server_pub`: Extracted public key

  - `ca`: Copy of CA bundle

**Note:** In the future, sub-levels for `register/` and `helper/` will be added under each CCF_ID.

### PKI Endpoints

**Root PKI Engine (`pki/`):**

- `root/generate/internal`: Generates root CA

- `config/urls`: Configures issuance and CRL URLs

- `root/sign-intermediate`: Signs intermediate CA

**Intermediate PKI Engine (`pki_int/`):**

- `intermediate/generate/internal`: Generates intermediate CA CSR

- `intermediate/set-signed`: Installs signed intermediate certificate

- `roles/my-ca`: Defines signing role with its policies

- `sign/my-ca`: Endpoint used by services to sign their CSRs

## Deployment Flow

### 1. Vault Initialization

**Execute once per Vault cluster:**

- In Docker: run `vault_prepare_certs.sh` script

- In Kubernetes: apply ConfigMap and Job from `helm/vault-job/vault-job.yaml`

**Expected result:**

- ✅ Root CA generated

- ✅ Intermediate CA generated and signed

- ✅ `my-ca` role configured

- ✅ CA bundle stored in `secret/ca`

### 2. CAPIF Instance Deployment

Each CAPIF instance (identified by unique CCF_ID):

1. **Helper starts** → Generates unique CCF_ID

2. **NGINX starts:**

   - Generates `server.key` and `server.csr`

   - Requests signing from Vault

   - Receives `server.crt`

   - Stores in `secret/capif/${CCF_ID}/nginx`

3. **Register starts:**

   - Generates registration certificate

   - Requests signing from Vault

4. **API Services start:**

   - Retrieve certificates from Vault using CCF_ID

   - Use public key for validation

## Advantages of New Architecture

### Security

1. **Private keys never transit**: each service generates its private key locally

2. **Principle of least privilege**: Vault only has CA role, doesn't manage service keys

3. **Isolation per CAPIF**: certificates separated by CCF_ID

### Scalability

1. **One Vault, multiple CAPIF**: total decoupling

2. **Namespace per instance**: `secret/capif/${CCF_ID}/`

3. **No collisions**: unique CCF_ID guarantees separation

### Operations

1. **Simplified rotation**: each CAPIF can regenerate its certificates independently

2. **Improved auditing**: can track which CAPIF requested which certificate

3. **Independent deployment**: a CAPIF can restart without affecting others

## Helm/Kubernetes Compatibility

The `helm/vault-job/vault-job.yaml` file has been significantly simplified:

**Main changes:**

- ❌ No longer generates certificates for specific services

- ✅ Only executes basic PKI setup (Root CA, Intermediate CA, signing role)

- ✅ Lighter: 144 lines removed

- ✅ ConfigMap contains a self-contained script that enables PKI engines, generates CAs and configures signing role

- ✅ Job runs only once when deploying Vault namespace

## Docker Compose Configuration

**Changes in `services/docker-compose-capif.yml`:**

**NGINX:**

- No longer mounts external certificate volumes

- Certificates are generated dynamically in `/etc/nginx/certs` at startup

- New environment variables: `VAULT_HOSTNAME`, `VAULT_PORT`, `VAULT_ACCESS_TOKEN`, `CAPIF_HOSTNAME`

**Helper:**

- Manages its own certificate directory internally

- `certs/` directory is automatically created by `app.py` at startup

- No longer requires external volumes for certificates

## Migration from Previous Architecture

### Migration steps:

1. **Backup existing certificates**: Export current certificates from Vault (optional, for history)

2. **Deploy new Vault**: Execute updated `vault_prepare_certs.sh` script

3. **Clean old certificates**: Remove certificate directories in NGINX and Register services

4. **Redeploy CAPIF services**: Execute `run.sh` so services automatically generate their new certificates

### Important considerations:

- ⚠️ **Downtime**: There will be a service interruption during migration

- ⚠️ **New CCF_ID**: Each instance will get a new unique CCF_ID

- ⚠️ **Regenerated certificates**: All service certificates will be new

- ✅ **No API changes**: CAPIF's public interface remains the same

- ✅ **Compatibility**: Existing clients will continue working after obtaining new CA

## Testing

**Recommended verifications:**

1. **Verify CA in Vault**: Query `secret/data/ca` with curl to confirm CA bundle is available

2. **Start CAPIF**: Execute startup script and observe logs from each service

3. **Verify NGINX certificate**: Use `openssl x509` inside container to inspect generated certificate

4. **Verify storage in Vault**: Obtain CCF_ID from Helper and query path `secret/capif/${CCF_ID}/nginx` in Vault

5. **Run test suite**: Run `run_capif_tests.sh` to validate complete functionality

## Troubleshooting

### Error: "Unable to retrieve CA certificate from Vault"

**Cause:** Vault has not been properly initialized with PKI infrastructure

**Solution:** Execute Vault preparation script (`vault_prepare_certs.sh`) inside container

### Error: "Failed to sign server certificate"

**Cause:** `my-ca` signing role is not configured or Vault token doesn't have sufficient permissions

**Solution:** Verify role exists and token has write permissions on `pki_int/sign/my-ca`

### Error: "Unable to retrieve CCF_ID from Helper"

**Cause:** Helper service hasn't completed initialization yet

**Solution:** Wait 30-60 seconds. Service scripts implement automatic retries with incremental delays

## References

### Modified Files (Branch OCF182-certs-generation)

- [`services/vault/vault_prepare_certs.sh`](https://labs.etsi.org/rep/ocf/capif/-/blob/staging/services/vault/vault_prepare_certs.sh) - PKI Setup

- [`services/nginx/nginx_prepare.sh`](https://labs.etsi.org/rep/ocf/capif/-/blob/staging/services/nginx/nginx_prepare.sh) - NGINX certs generation

- [`services/register/register_prepare.sh`](https://labs.etsi.org/rep/ocf/capif/-/blob/staging/services/register/register_prepare.sh) - Register certs generation

- [`services/helper/helper_service/app.py`](https://labs.etsi.org/rep/ocf/capif/-/blob/staging/services/helper/helper_service/app.py) - Superadmin cert generation

- [`services/TS29222_CAPIF_API_Invoker_Management_API/prepare_invoker.sh`](https://labs.etsi.org/rep/ocf/capif/-/blob/staging/services/TS29222_CAPIF_API_Invoker_Management_API/prepare_invoker.sh) - Cert consumption

- [`services/TS29222_CAPIF_API_Provider_Management_API/prepare_provider.sh`](https://labs.etsi.org/rep/ocf/capif/-/blob/staging/services/TS29222_CAPIF_API_Provider_Management_API/prepare_provider.sh) - Cert consumption

- [`services/TS29222_CAPIF_Security_API/prepare_security.sh`](https://labs.etsi.org/rep/ocf/capif/-/blob/staging/services/TS29222_CAPIF_Security_API/prepare_security.sh) - Cert consumption

- [`helm/vault-job/vault-job.yaml`](https://labs.etsi.org/rep/ocf/capif/-/blob/staging/helm/vault-job/vault-job.yaml) - Kubernetes Job

- [`services/docker-compose-capif.yml`](https://labs.etsi.org/rep/ocf/capif/-/blob/staging/services/docker-compose-capif.yml) - Docker Configuration

### Related Documentation

- [HashiCorp Vault PKI Secrets Engine](https://developer.hashicorp.com/vault/docs/secrets/pki)

- [OpenSSL CSR Generation](https://www.openssl.org/docs/man1.1.1/man1/openssl-req.html)

- [CAPIF Specification TS 29.222](https://www.3gpp.org/DynaReport/29222.htm)

---

**Document Version**: 1.0  

**Date**: February 2026  

**Branch**: OCF182-certs-generation  

**Author**: CAPIF Team

      - Event Filter: ./event-filter/event-filter.md

      - Dynamic Configuration: ./configuration/configuration.md

      - Event Reporting Information: ./event-req/event-req.md

      - Vault Certificate Management: ./vault/vault.md

  - SDK:

      - Introduction: ./sdk/sdk_introduction.md

      - Requirements: ./sdk/sdk_requirements.md