Improve MEC Sandbox deployment pipeline and repository sanitation in PyInfra

---

## Automated Deployment Workflow

The diagram below illustrates the automated deployment workflow from setup and `.env` configuration to validation and execution:

```mermaid

flowchart LR

    subgraph SETUP ["1. Setup & Config"]

        direction TB

        S1["run ./setup.sh"] --> S2["source pyinfra-venv/bin/activate"]

        S2 --> S3["edit .env (K8S_MASTERS, MEC_HOST, OAuth)"]

    end

    subgraph DEPLOY ["2. Execute Deployment"]

        direction TB

        D1["pyinfra inventory.py deploy.py"] --> D2{"OAuth & Config Valid?"}

        D2 -- "No" --> ERR["Halt with validation error"]

        D2 -- "Yes" --> D3["Prompt for Sudo Passwords (in-memory)"]

    end

    subgraph ENGINE ["3. Automated Provisioning"]

        direction TB

        E1["System & Kubernetes (kubeadm)"] --> E2["Build & Deploy Microservices"]

        E2 --> E3(["MEC Sandbox Live"])

    end

    S3 --> D1

    D3 --> E1

    style S1 fill:#1f6feb,stroke:#388bfd,color:#ffffff

    style D1 fill:#238636,stroke:#2ea043,color:#ffffff

    style E3 fill:#8957e5,stroke:#a371f7,color:#ffffff

    style ERR fill:#da3633,stroke:#f85149,color:#ffffff

```

---

## Prerequisites

Before deploying, ensure your target machine(s) meet the following requirements:

Run the automated setup script to verify Python 3, create an isolated virtual environment (`pyinfra-venv`), and install all required deployment dependencies:

```bash

cd pyinfra

cd ~/etsi-mec-sandbox/pyinfra

./setup.sh

```

The first time you run `./setup.sh`, it generates a `.env` configuration file from `.env.example` and pauses so you can enter your settings.

Open `.env` in your text editor and configure the following required fields:

- **`K8S_MASTERS`:** Mandatory target host for the Kubernetes control plane (exactly 1 master node is supported; e.g., `localhost` for local deployments, or `ubuntu@192.168.1.10` for remote servers).

- **`K8S_MASTERS`:** Mandatory target host for the Kubernetes control plane (<strong style="color: #d9822b;">exactly 1 master node is supported</strong>; e.g., `localhost` for local deployments, or `ubuntu@192.168.1.10` for remote servers).

- **`K8S_WORKERS`:** Optional comma-separated list of worker node IPs/hostnames. Leave blank (`""`) for single-machine deployments.

- **`MEC_HOST_ADDRESS`:** The routable IP address or domain name where the MEC Sandbox frontend will be accessible (e.g., `127.0.0.1`, `192.168.1.100`, or `mec.example.com`).

- **OAuth Provider Credentials:** Provide valid OAuth secrets for **GitHub** (`GITHUB_CLIENT_ID`, `GITHUB_CLIENT_SECRET`), **GitLab**, or both. Unconfigured providers are automatically disabled in the platform configuration.

> [!IMPORTANT]

> **Do not set both `K8S_MASTERS` and `K8S_WORKERS` to `localhost`.**  

> ! IMPORTANT</br>

> <strong style="color: #e51400;">DO NOT set both <code>K8S_MASTERS</code> and <code>K8S_WORKERS</code> to <code>localhost</code>.</strong>  

> A single machine cannot act as both an independent Kubernetes master and worker node. For an all-in-one sandbox on your local machine, set `K8S_MASTERS="localhost"` and leave `K8S_WORKERS=""`.

### Step 3: Run the Deployment

## Authentication & Sudo Passwords

For security, **sudo passwords are never stored in config files or environment variables.**

For security, <strong style="color: #e51400;">sudo passwords are NEVER stored in config files or environment variables.</strong>

When you launch `pyinfra inventory.py deploy.py`:

1. **Prompted Once at Startup:** PyInfra will prompt you in the terminal for your sudo password:

1. **Prompted Once at Startup:** PyInfra will <strong style="color: #28a745;">prompt you interactively once at startup</strong> in the terminal for your sudo password:

   ```text

   Enter sudo password for K8S_MASTERS node(s) (press Enter for passwordless sudo):

   ```

### Option B: Remote / Multi-Node Kubernetes Cluster

To deploy across multiple remote servers:

1. **Configure Targets in `.env`:** Specify target remote hosts in mandatory `<username>@<ip>` format (note: exactly 1 master node is supported for the control plane):

1. **Configure Targets in `.env`:** Specify target remote hosts in <strong style="color: #d9822b;">mandatory <code>&lt;username&gt;@&lt;ip&gt;</code> format</strong> (note: <strong style="color: #d9822b;">exactly 1 master node is supported</strong> for the control plane):

   ```env

   K8S_MASTERS="ubuntu@192.168.1.10"

   K8S_WORKERS="ubuntu@192.168.1.11,admin@192.168.1.12"

## Resuming Interrupted Deployments

The deployment process is **idempotent and checkpointed**:

The deployment process is <strong style="color: #28a745;">idempotent and checkpointed</strong>:

- Long-running stages (such as compiling `meepctl` binaries and packaging container images) create checkpoint markers automatically.

- If your network disconnects or an execution is interrupted, simply re-run `pyinfra inventory.py deploy.py`.

- The installer will skip all completed stages and resume immediately from the last checkpoint without restarting from scratch.

---

## Redeployment & Troubleshooting

### Full Redeployment of the Sandbox

The deployment lifecycle creates checkpoint markers in `~/.meep/` to track completed stages:

- `~/.meep/.01_secrets_configured`

- `~/.meep/.02_deps_deployed`

- `~/.meep/.03_binaries_built`

- `~/.meep/.04_images_dockerized`

- `~/.meep/.05_scenarios_imported`

To perform a <strong style="color: #28a745;">full clean redeployment</strong> of the sandbox, <strong style="color: #d9822b;">remove these checkpoint files</strong> and re-run the deployment command:

```bash

rm -f ~/.meep/.01_secrets_configured ~/.meep/.02_deps_deployed ~/.meep/.03_binaries_built ~/.meep/.04_images_dockerized ~/.meep/.05_scenarios_imported

source pyinfra-venv/bin/activate

pyinfra inventory.py deploy.py -y

```

### Single Component Redeployment & Troubleshooting

For redeploying a single microservice component or diagnosing specific container issues, refer to the official troubleshooting guide:

- [MEEPCTL Troubleshooting & Single Component Guide](https://labs.etsi.org/rep/mec/etsi-mec-sandbox-frontend/-/blob/STF_685/guides/meepctl-troubleshooting.md?ref_type=heads)

# Kubernetes Master Setup

if "k8s_masters" in host.groups:

    # # System Configuration

    # System Configuration

    # local.include("tasks/system/common.py")

    # local.include("tasks/system/kernel.py")

    # # Container Runtime

    # local.include("tasks/k8s_cluster/cni_calico.py")

    # local.include("tasks/k8s_cluster/helm.py")

    # Dev Environment & Sandbox

    install_dev_env = host.data.get('install_dev_env', True)

    # # Dev Environment & Sandbox

    # install_dev_env = host.data.get('install_dev_env', True)

    install_mec_sandbox = host.data.get('install_mec_sandbox', True)

    # if install_dev_env:

    #     local.include("tasks/apps/dev_env.py")

    "tar",

    "python3",

    "python3-pip",

    "python3-yaml",

    "python3-ruamel.yaml",

    "acl",

]

import os as __os

from lib.config_helpers import (

    get_k8s_masters as __get_k8s_masters,

    get_k8s_workers as __get_k8s_workers,

    validate_k8s_hosts as __validate_k8s_hosts,

    get_master_sudo_password as __get_master_sudo_password,

    get_worker_sudo_password as __get_worker_sudo_password,

)

# 1. Host Resolution & Validation:

#    - Converts "localhost"/"127.0.0.1" to PyInfra's local executor tuple ("@local", user).

#    - Validates that "@local" is not assigned to both master and worker groups, as a single

#      OS instance cannot act as both an independent control-plane master and worker node.

__master_hosts = __get_k8s_masters()

__worker_hosts = __get_k8s_workers()

__validate_k8s_hosts(__master_hosts, __worker_hosts)

# 2. Interactive Sudo Credential Acquisition:

#    - Prompts the user exactly once during inventory compilation.

#    - Master and worker nodes can use independent sudo credentials.

#    - In non-interactive/CI pipelines, defaults to None without hanging.

__master_sudo_password = __get_master_sudo_password()

__worker_sudo_password = __get_worker_sudo_password() if __worker_hosts else None

# 3. PyInfra Host Group Definitions:

#    - PyInfra caches the host data dictionary (ssh_user, sudo_password) in memory.

#    - During deployment, any task invoked with `_sudo=True` automatically uses the stored

#      sudo_password to authenticate via `sudo -S` non-interactively.

k8s_masters = [

    (host_addr, {"ssh_user": ssh_user, "sudo_password": __master_sudo_password})

    for host_addr, ssh_user in __master_hosts

]

k8s_workers = [

    (host_addr, {"ssh_user": ssh_user, "sudo_password": __worker_sudo_password})

    for host_addr, ssh_user in __worker_hosts

]

"""

Declarative PyInfra Inventory for ETSI MEC Sandbox Deployment.

All host resolution, validation, and credential acquisition logic is modularized

in lib/config_helpers.py:

- For remote targets (<user>@<host>), automatically re-uses the provided sudo password

  as the SSH fallback password so users are only prompted once.

- For localhost (@local), no SSH password is needed.

- If SSH keys or passwordless sudo are configured, pressing Enter defaults passwords to None.

"""

from lib.config_helpers import get_k8s_inventory

k8s_masters, k8s_workers = get_k8s_inventory()

"""

Configuration helpers for PyInfra deployment of ETSI MEC Sandbox.

Organized into clean functional sections by comment banners:

1. Environment & Secret Helpers

2. Kubernetes Host, Inventory & Sudo Helpers

3. MEC Sandbox & OAuth Configuration Helpers

"""

import os

import sys

import getpass

from lib.scripts.verify_oauth import verify_github_oauth, verify_gitlab_oauth

# ======================================================================

# 1. ENVIRONMENT & SECRET HELPERS

# ======================================================================

# Automatically load .env variables from pyinfra root before reading any configuration

_env_path = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), ".env")

except ImportError:

    pass

def is_valid_secret(val, placeholders=None):

    """Checks if a secret or config variable is present and not a default placeholder."""

    if not val:

        return False

    return True

def is_interactive():

    """Returns True if running in an interactive terminal without CI flags."""

    # Check if standard input is attached to a TTY (terminal)

        return False

    return True

# ======================================================================

# 2. KUBERNETES HOST, INVENTORY & SUDO HELPERS

# ======================================================================

def _parse_host_list(raw_str):

    """

    Parses a comma-separated host string into PyInfra host tuples: (address, ssh_user).

            hosts.append((addr_clean, user_clean))

    return hosts

def get_k8s_masters():

    """

    Retrieves K8S_MASTERS from environment or .env.

    Mandatory: if unset/empty in interactive mode, prompts the user.

def get_k8s_inventory():

    """

    raw = os.environ.get("K8S_MASTERS")

    if not raw or not str(raw).strip():

        if is_interactive():

            raw = input("Enter IP or hostname for K8S_MASTERS [default: localhost]: ").strip() or "localhost"

        else:

            raise ValueError("K8S_MASTERS is mandatory and must be set in environment or .env file.")

    return _parse_host_list(raw)

def get_k8s_workers():

    """Retrieves K8S_WORKERS from environment or .env."""

    raw = os.environ.get("K8S_WORKERS", "")

    return _parse_host_list(raw)

    Resolves, validates, and builds the declarative PyInfra inventory tuples

    for k8s_masters and k8s_workers.

def validate_k8s_hosts(masters, workers):

    - Validates that K8S_MASTERS contains exactly 1 master node and is not co-located with workers on @local.

    - For remote targets (<user>@<host>), automatically re-uses the single provided password

      for both ssh_password and sudo_password so users are only prompted once.

    - For localhost (@local), no SSH password is needed.

    - If SSH keys or passwordless sudo are configured, pressing Enter defaults passwords to None.

    """

    Validates that K8S_MASTERS and K8S_WORKERS do not both contain localhost (@local).

    A single machine cannot act as both an independent master and worker node.

    """

    # Extract addresses from (address, ssh_user) tuples

    masters = get_k8s_masters()

    workers = get_k8s_workers()

    # 1. Validation

    master_addrs = [addr for addr, _ in masters]

    if len(masters) != 1:

        raise ValueError(

        )

    worker_addrs = [addr for addr, _ in workers]

    # Kubernetes control plane and worker daemons conflict when deployed as separate nodes

    # on the same physical host or local VM instance.

    if "@local" in master_addrs and "@local" in worker_addrs:

        raise ValueError(

            "K8S_MASTERS and K8S_WORKERS cannot both be set to localhost (@local). "

            "A single machine cannot act as both an independent master and worker node."

        )

def get_master_sudo_password():

    """

    Interactively prompts for K8S_MASTERS sudo password during execution.

    Per strict security policy, never reads from .env or uses a hardcoded fallback.

    """

    # By prompting via getpass only in interactive sessions and never falling back to

    # hardcoded strings or .env variables, we prevent credential leaks in git/env files.

    # 2. Credential acquisition (single unified prompt)

    master_pw = None

    worker_pw = None

    if is_interactive():

        prompted = getpass.getpass("Enter sudo password for K8S_MASTERS node(s) (press Enter for passwordless sudo): ").strip()

        return prompted if prompted else None

    return None

        prompted_m = getpass.getpass("Enter password for K8S_MASTERS node(s) (for SSH/sudo, or press Enter if using SSH keys/passwordless sudo): ").strip()

        master_pw = prompted_m if prompted_m else None

        if workers:

            prompted_w = getpass.getpass("Enter password for K8S_WORKERS node(s) (for SSH/sudo, or press Enter if using SSH keys/passwordless sudo): ").strip()

            worker_pw = prompted_w if prompted_w else None

def get_worker_sudo_password():

    """

    Interactively prompts for K8S_WORKERS sudo password during execution.

    Called only when worker nodes are present in the inventory.

    """

    if is_interactive():

        prompted = getpass.getpass("Enter sudo password for K8S_WORKERS node(s) (press Enter for passwordless sudo): ").strip()

        return prompted if prompted else None

    return None

    # 3. Build PyInfra host tuples

    def _build_host_tuple(host_addr, ssh_user, pw):

        data = {"ssh_user": ssh_user, "sudo_password": pw}

        if host_addr != "@local" and pw:

            data["ssh_password"] = pw

        return (host_addr, data)

    k8s_masters = [_build_host_tuple(addr, user, master_pw) for addr, user in masters]

    k8s_workers = [_build_host_tuple(addr, user, worker_pw) for addr, user in workers]

    return k8s_masters, k8s_workers

def get_sudo_password():

    """Alias to get_master_sudo_password for backward compatibility."""

    return get_master_sudo_password()

def get_target_user_and_home():

    """

    target_home = f"/home/{target_user}"

    return target_user, target_home

def get_k8s_masters():

    """

    Retrieves K8S_MASTERS from environment or .env.

    Mandatory: if unset/empty in interactive mode, prompts the user.

    """

    raw = os.environ.get("K8S_MASTERS")

    if not raw or not str(raw).strip():

        if is_interactive():

            raw = input("Enter IP or hostname for K8S_MASTERS [default: localhost]: ").strip() or "localhost"

        else:

            raise ValueError("K8S_MASTERS is mandatory and must be set in environment or .env file.")

    return _parse_host_list(raw)

def get_k8s_workers():

    """Retrieves K8S_WORKERS from environment or .env."""

    raw = os.environ.get("K8S_WORKERS", "")

    return _parse_host_list(raw)

# ======================================================================

# 3. MEC SANDBOX & OAUTH CONFIGURATION HELPERS

# ======================================================================

def get_mec_host_address():

    """

    Retrieves and validates the MEC Sandbox host address from environment or interactive prompt.

        raise ValueError("MEC_HOST_ADDRESS cannot be empty.")

    return mec_host_address

def get_oauth_config():

    """

    Retrieves and validates GitHub and/or GitLab OAuth credentials.

    configured_providers = []

    if github_enabled:

        valid, msg = verify_github_oauth(gh_id_raw, gh_sec_raw)

        if not valid:

            raise ValueError(f"GitHub OAuth credentials failed live API check: {msg}")

        print(f"[OAuth Check] {msg}")

        configured_providers.append("github")

    if gitlab_enabled:

        valid, msg = verify_gitlab_oauth(gl_id_raw, gl_sec_raw)

        if not valid:

            raise ValueError(f"GitLab OAuth credentials failed live API check: {msg}")

        print(f"[OAuth Check] {msg}")

        configured_providers.append("gitlab")

    return {

        "gitlab_client_secret": gl_sec_raw if gitlab_enabled else "",

        "configured_providers": configured_providers,

    }

__all__ = [

    "is_valid_secret",

    "is_interactive",

    "_parse_host_list",

    "get_k8s_masters",

    "get_k8s_workers",

    "validate_k8s_hosts",

    "get_master_sudo_password",

    "get_worker_sudo_password",

    "get_sudo_password",

    "get_target_user_and_home",

    "get_mec_host_address",

    "get_oauth_config",

]