Release TF-SDK 2.0.0

# SDK verification and release gates

The default pipeline validates Python 3.12 using a hashed dependency/tool lock.

Run `python ci/verify.py SUITE` for lint, build, tests, sdist, charts and typing.

Build first; subsequent suites consume that wheel, not an editable checkout.

Tests select contract, common and offline location cases explicitly. Live NEF

cases are excluded and require separate acceptance evidence.

JUnit, focused contract branch coverage (81% minimum), environment inventories,

distribution hashes and packaged chart renders are retained as CI artifacts.

Helm 3.18.6 is downloaded with its official archive checksum. Python 3.10,

3.11 and 3.13 run in scheduled and release pipelines using digest-pinned images.

Update tools.in, requirements.lock, runtime.lock and toolchain.json together.

Locks were compiled with `uv pip compile pyproject.toml ci/tools.in --universal --python-version 3.10

--generate-hashes` (omit ci/tools.in for runtime.lock). Review dependency updates.

`python ci/security.py secrets` rejects new findings. The reviewed fingerprint

file contains hashes and reasons for synthetic schema examples, generated field

maps and the Helm checksum; only this hash inventory is excluded from scanning.

Changing a value or moving it to another file requires a new review. Access-token

fixtures use an explicit example value. Never place real credentials in fixtures.

Dependency audit produces vulnerability JSON and CycloneDX SBOM; tag and nightly

runs must pass. Audit service failures cannot produce a clean result.

Source, sdist and wheel are checked before publication. Branch pushes cannot

publish packages. Release promotion reuses the built and tested bytes; never

rebuild a package under an existing version.

The candidate version is 2.0.0rc5. Review the release scope and applicable E2E

evidence, then create a protected matching release tag when publication is

approved. After required CI jobs pass, publish-package automatically uploads

the verified artifacts using CI_JOB_TOKEN. No manual job, qualification file,

extra pipeline IDs or exception variables are required. Tag/version/source

and checksum guards remain mandatory.

An automatic informational release-qualification.json preserves actual SDK and

linked downstream test reports. Missing reports remain visible; applicable E2E

results are reviewed in the release MR. See [the release procedure](RELEASE.md).

GitLab Runner downloads artifacts before before_script. Its initial TMPDIR is the

existing project directory; individual verification commands then create and use

.cache/ci/tmp. This keeps temporary files off tmpfs without breaking downloads.

Python support is bounded to >=3.10,<3.14, matching the classifiers and tested

matrix; the current pinned pydantic-core cannot build with Python 3.14. Release

reports record available evidence, not signed attestations or a claimed SLSA

level. The release owner judges whether the tested scope is sufficient for the

release before creating its protected publication tag.

Consumer qualification follows a protected SRM candidate branch. GitLab fixes

the downstream checkout when creating that pipeline, and its receipt records

the exact consumer revision. Advancing SRM can requalify the same immutable SDK

artifact without changing the package or inventing a new SDK version. Use

RUN_EXTENDED=1 to run the full Python/security matrix in a web pipeline too.

The downstream trigger disables inheritance of SDK default variables and passes

only the explicit SDK artifact coordinates. Job-only paths such as TMPDIR must

be established by the owning project. Candidate rc1 exposed this boundary issue

and was not published; rc2 retains a new immutable tag instead of moving rc1.

Packaged container chart checks render canonical CPU millicores and decimal MB

requests, including omission and absence of invented limits. Provider regressions

exercise the same mapping through activation and update.

# Release an SDK package

1. Review the release scope and merge the stable source/version into protected

   main through its release MR. The merge is the publication approval.

2. The main pipeline builds the wheel and source distribution once, verifies

   those bytes, runs the Python matrix, dependency audit and linked SRM consumer.

3. After required checks pass, publish-package creates v2.0.0 for package

   2.0.0 and uploads those same artifacts to the existing GitLab Python registry.

   The job-token tag push starts no second pipeline. GitLab Release pages remain

   with the release owner. Explicit protected candidate/tag pipelines remain

   available for trials and the existing publication path.

No manual job, qualification file, pipeline-ID input, extra approval form or

exception flag is required. Develop and MR pipelines do not publish packages.

## One-time project setting

Enable Settings → CI/CD → Job token permissions → Allow Git push requests to

the repository for this SDK project. GitLab calls the setting

ci_push_repository_for_job_token_allowed. This allows same-project job tokens

to push with their initiating user's repository permissions; it applies to

the project's jobs, not only this publisher. No new long-lived token is needed.

Keep branch/tag protections and cross-project pushes unchanged. The main merge

must be performed by a Maintainer allowed to create the protected release tag.

The publisher automatically saves release-qualification.json with artifact

identity, the GitLab publication job URL, actual SDK JUnit summary and linked

downstream pipeline/test results. It uses the current public project APIs without

a new token. Missing or inaccessible reports are recorded as unavailable and

cannot add a new publication gate. External E2E runs remain visible in their

own pipelines and are reviewed in the release MR; this collector cannot certify

application, NEF or device acceptance from SDK/consumer results.

The existing required jobs and protected tag/version/source/checksum checks

remain enforced. Report collection changes no test outcome. If evidence is

incomplete, the release owner decides whether to wait or merge and can explain

the decision in the existing MR. No separate exception

workflow is needed.

If a required job fails, fix/retry it or use an explicitly approved targeted CI

change through the normal review process.

For upload failures, retry the same job with the original artifacts. The publisher

checks registry hashes, skips identical files and uploads only missing files.

A different checksum fails. A tag at another commit is never moved or replaced;

later main work at that already-tagged version skips publication. A new source

release needs a new package version.

A source, version or rebuilt-artifact change needs a fresh version/tag and checks.

## Order across repositories

Verify the candidate wheel directly in SRM before registry publication. Keep

the working protected consumer ref during the first final SDK publication;

do not switch that trigger to a consumer whose setup requires the unpublished

package. Applicable E2E evidence is reviewed here, while full final-platform

acceptance follows the normal SRM image built with the published final package.

After the SDK main pipeline creates the tag and publishes the package, verify

the registry files against that pipeline's build checksums. SRM maintainers then

update pyproject.toml and uv.lock together from the actual registry, run normal

locked checks and publish their image. Update chart/profile references, run

the applicable OEG-driven E2E and review the platform release MRs. The release

owner creates the agreed final platform tags and GitLab Releases. SDK package

availability must precede SRM's dependency installation, even when release

pages and the coordinated platform announcement are created later.

"""Check release identity and qualification before publishing verified artifacts."""

import argparse

import json

import os

import re

import subprocess

import sys

import zipfile

from email.parser import BytesParser

from urllib.parse import urlencode

from urllib.request import Request, urlopen

from packaging.version import Version

from reports import collect_reports

from verify import DIST, OUT, ROOT, sha256, write_json

def check_tag(tag, version, protected):

    if not re.fullmatch(r"v2\.0\.\d+(?:rc[1-9][0-9]*)?", tag):

        raise ValueError("Use an immutable v2.0.X or v2.0.XrcN release tag")

    if Version(tag[1:]) != Version(version):

        raise ValueError("Release tag and package version differ")

    if protected != "true":

        raise ValueError("Publication requires a protected release tag")

def release_context(version):

    tag = os.getenv("CI_COMMIT_TAG", "")

    if tag:

        return tag, False

    if (

        os.getenv("CI_COMMIT_BRANCH") != "main"

        or os.getenv("CI_COMMIT_REF_PROTECTED") != "true"

        or os.getenv("CI_PIPELINE_SOURCE") != "push"

    ):

        raise ValueError("Automatic publication requires a protected main push")

    if Version(version).is_prerelease:

        raise ValueError("Use a stable package version for automatic main publication")

    return f"v{version}", True

def ensure_main_tag(tag, revision, *, create=True):

    ref = f"refs/tags/{tag}"

    raw = subprocess.check_output(

        ["git", "ls-remote", "--tags", "origin", ref, ref + "^{}"], text=True

    )

    refs = {name: sha for sha, name in (line.split() for line in raw.splitlines())}

    existing = refs.get(ref + "^{}", refs.get(ref))

    if existing:

        return existing == revision  # A later main merge cannot move a released version.

    if not create:

        return True

    try:

        subprocess.run(

            ["git", "push", "origin", f"{revision}:{ref}"], check=True, capture_output=True

        )

    except subprocess.CalledProcessError:

        raise ValueError(

            "Tag creation failed; enable same-project job-token Git pushes "

            "and use a Maintainer-triggered pipeline"

        ) from None

    return True

def registry_json(path):

    base = f"{os.environ['CI_API_V4_URL']}/projects/{os.environ['CI_PROJECT_ID']}"

    request = Request(base + path, headers={"JOB-TOKEN": os.environ["CI_JOB_TOKEN"]})

    with urlopen(request, timeout=30) as response:

        return json.load(response)

def pending_artifacts(version):

    query = urlencode(

        {

            "package_type": "pypi",

            "package_name": "openop-tfsdk",

            "package_version": version,

            "per_page": 100,

        }

    )

    existing = {}

    for package in registry_json("/packages?" + query):

        if package["name"] != "openop-tfsdk" or package["version"] != version:

            continue

        for item in registry_json(f"/packages/{package['id']}/package_files"):

            name, checksum = item["file_name"], item["file_sha256"]

            if name in existing and existing[name] != checksum:

                raise ValueError("Registry contains conflicting artifact checksums")

            existing[name] = checksum

    pending = []

    for path in sorted(DIST.iterdir()):

        if path.name not in existing:

            pending.append(path)

        elif existing[path.name] != sha256(path):

            raise ValueError("Published artifact differs; never overwrite a released version")

    return pending

def record_publication(receipt, status):

    receipt["publication_status"] = status

    write_json(OUT / "release-check.json", receipt)

    qualification = OUT / "release-qualification.json"

    write_json(qualification, {**json.loads(qualification.read_text()), **receipt})

def verify_release():

    manifest = json.loads((OUT / "manifest.json").read_text())

    for item in manifest["artifacts"]:

        if sha256(DIST / item["name"]) != item["sha256"]:

            raise ValueError("Artifact changed after verification")

    wheel = next(DIST.glob("*.whl"))

    with zipfile.ZipFile(wheel) as archive:

        name = next(n for n in archive.namelist() if n.endswith(".dist-info/METADATA"))

        metadata = BytesParser().parsebytes(archive.read(name))

    if metadata["Name"] != "openop-tfsdk":

        raise ValueError("Unexpected distribution name")

    version = metadata["Version"]

    tag, automatic = release_context(version)

    check_tag(tag, version, os.getenv("CI_COMMIT_REF_PROTECTED"))

    if manifest["source_sha"] != os.environ["CI_COMMIT_SHA"]:

        raise ValueError("Build and tag source revisions differ")

    # Tag pipelines may fetch only a detached tag; refresh the allowed ancestry refs.

    subprocess.run(["git", "fetch", "origin", "+refs/heads/*:refs/remotes/origin/*"], check=True)

    branches = (

        ["origin/main"]

        if automatic

        else ["origin/develop", "origin/main", "origin/refactor/tfsdk-architecture-alignment"]

    )

    if not any(

        subprocess.run(

            ["git", "merge-base", "--is-ancestor", "HEAD", ref], capture_output=True

        ).returncode

        == 0

        for ref in branches

    ):

        raise ValueError("Tagged commit is outside the reviewed integration lines")

    write_json(

        OUT / "release-check.json",

        {

            "tag": tag,

            "version": version,

            "source_sha": manifest["source_sha"],

            "wheel_sha256": sha256(wheel),

            "release_type": "candidate" if Version(version).is_prerelease else "final",

            "publication_job_url": os.getenv("CI_JOB_URL"),

            "approval_source": "protected_main_merge" if automatic else "protected_tag",

        },

    )

    receipt = json.loads((OUT / "release-check.json").read_text())

    write_json(OUT / "release-qualification.json", {**receipt, **collect_reports()})

    return receipt

def main():

    parser = argparse.ArgumentParser()

    parser.add_argument("action", choices=["check", "publish"])

    args = parser.parse_args()

    os.chdir(ROOT)

    receipt = verify_release()

    if args.action == "publish":

        if receipt["approval_source"] == "protected_main_merge" and not ensure_main_tag(

            receipt["tag"], receipt["source_sha"], create=False

        ):

            record_publication(receipt, "skipped_existing_release_tag")

            print("This version already has a release tag; no tag or package was replaced")

            return

        pending = pending_artifacts(receipt["version"])

        if receipt["approval_source"] == "protected_main_merge" and not ensure_main_tag(

            receipt["tag"], receipt["source_sha"]

        ):

            raise ValueError("Release tag changed during publication; no package was uploaded")

        if not pending:

            record_publication(receipt, "already_published_matching_checksums")

            print("Verified artifacts are already published")

            return

        environment = dict(

            os.environ, TWINE_USERNAME="gitlab-ci-token", TWINE_PASSWORD=os.environ["CI_JOB_TOKEN"]

        )

        url = f"{os.environ['CI_API_V4_URL']}/projects/{os.environ['CI_PROJECT_ID']}/packages/pypi"

        subprocess.run(

            [

                sys.executable,

                "-m",

                "twine",

                "upload",

                "--repository-url",

                url,

                *[str(p) for p in pending],

            ],

            env=environment,

            check=True,

        )

        record_publication(receipt, "published")

if __name__ == "__main__":

    main()

"""Collect informational GitLab test reports without another release gate."""

import json

import os

from urllib.request import urlopen

def collect_reports():

    report = {

        "pipeline_url": os.getenv("CI_PIPELINE_URL"),

        "sdk_tests": None,

        "downstreams": [],

        "unavailable": [],

        "external_e2e": "Review applicable acceptance results in the release MR.",

    }

    base = os.getenv("CI_API_V4_URL", "").rstrip("/")

    project = os.getenv("CI_PROJECT_ID", "")

    pipeline = os.getenv("CI_PIPELINE_ID", "")

    if not base or not project or not pipeline:

        report["unavailable"].append("Missing GitLab pipeline context")

        return report

    def read(path):

        try:

            # The current projects expose these reports publicly. No new token.

            with urlopen(f"{base}/{path}", timeout=5) as response:

                return json.load(response)

        except (OSError, ValueError) as error:

            report["unavailable"].append(f"{path}: {type(error).__name__}")

            print(f"CI report unavailable: {path} ({type(error).__name__})")

            return None

    prefix = f"projects/{project}/pipelines/{pipeline}"

    report["sdk_tests"] = read(prefix + "/test_report_summary")

    page = 1

    while True:

        bridges = read(prefix + f"/bridges?per_page=100&page={page}")

        if not isinstance(bridges, list):

            if bridges is not None:

                report["unavailable"].append("Unexpected downstream pipeline response")

            break

        for bridge in bridges:

            try:

                downstream = bridge.get("downstream_pipeline")

                item = {

                    "job": bridge["name"],

                    "status": bridge["status"],

                    "pipeline": downstream,

                }

                if downstream:

                    path = f"projects/{downstream['project_id']}/pipelines/{downstream['id']}"

                    item["tests"] = read(path + "/test_report_summary")

                report["downstreams"].append(item)

            except (AttributeError, KeyError, TypeError):

                report["unavailable"].append("Unexpected downstream pipeline entry")

        if len(bridges) < 100:

            break

        page += 1

    return report