Commit 3a40fe1b authored by Konstantinos Togias's avatar Konstantinos Togias
Browse files

ci: tag and publish verified stable releases after main merges

parent 737eaa00
Loading
Loading
Loading
Loading
+5 −2
Changes for .gitlab-ci.yml: 5 added lines, 2 removed lines.
Original line number Diff line number Diff line
@@ -115,7 +115,7 @@ dependency-audit:
  script:
  - python ci/security.py dependencies
  rules:
  - if: $CI_COMMIT_TAG || $CI_PIPELINE_SOURCE == "schedule" || $RUN_EXTENDED == "1"
  - if: $CI_COMMIT_TAG || ($CI_COMMIT_BRANCH == "main" && $CI_COMMIT_REF_PROTECTED == "true" && $CI_PIPELINE_SOURCE == "push") || $CI_PIPELINE_SOURCE == "schedule" || $RUN_EXTENDED == "1"
  - when: manual
    allow_failure: true
  artifacts:
@@ -146,7 +146,7 @@ python-compatibility:
  - job: build-package
    artifacts: true
  rules:
  - if: $CI_COMMIT_TAG || $CI_PIPELINE_SOURCE == "schedule" || $RUN_EXTENDED == "1"
  - if: $CI_COMMIT_TAG || ($CI_COMMIT_BRANCH == "main" && $CI_COMMIT_REF_PROTECTED == "true" && $CI_PIPELINE_SOURCE == "push") || $CI_PIPELINE_SOURCE == "schedule" || $RUN_EXTENDED == "1"
  script:
  - python ci/verify.py tests
  artifacts:
@@ -172,6 +172,9 @@ publish-package:
    - job: python-compatibility
    - job: srm-consumer
  rules:
    - if: '$CI_COMMIT_BRANCH == "main" && $CI_COMMIT_REF_PROTECTED == "true" && $CI_PIPELINE_SOURCE == "push"'
      when: on_success
      allow_failure: false
    - if: '$CI_COMMIT_TAG =~ /^v2[.]0[.][0-9]+(rc[1-9][0-9]*|-release)?$/ && $CI_COMMIT_REF_PROTECTED == "true"'
      when: on_success
      allow_failure: false
+25 −14
Changes for ci/RELEASE.md: 25 added lines, 14 removed lines.
Original line number Diff line number Diff line
# Release an SDK package

1. Review the release scope and applicable OEG-driven acceptance results, and
   merge the intended source/version through the release MR.
2. The release owner creates its matching protected tag. The existing platform
   convention gives v2.0.0-release for Python package version 2.0.0; plain
   v2.0.0 and candidate v2.0.0rcN remain supported.
   Creating the tag approves publication after successful checks.
   CI builds the wheel and source distribution once, tests those bytes and runs
   the linked SRM consumer. The release matrix and dependency audit also run.
3. After required CI jobs pass, **publish-package** automatically uploads the
   tested artifacts to the project's GitLab Python package registry.
1. Review the release scope and merge the stable source/version into protected
   main through its release MR. The merge is the publication approval.
2. The main pipeline builds the wheel and source distribution once, verifies
   those bytes, runs the Python matrix, dependency audit and linked SRM consumer.
3. After required checks pass, publish-package creates v2.0.0-release for package
   2.0.0 and uploads those same artifacts to the existing GitLab Python registry.
   The job-token tag push starts no second pipeline. GitLab Release pages remain
   with the release owner. Explicit protected candidate/tag pipelines remain
   available for trials and the existing publication path.

No manual job, qualification file, pipeline-ID input, extra approval form or
exception flag is required. Merging the release MR integrates source; branch
pipelines continue to avoid package uploads.
exception flag is required. Develop and MR pipelines do not publish packages.

## One-time project setting

Enable Settings → CI/CD → Job token permissions → Allow Git push requests to
the repository for this SDK project. GitLab calls the setting
ci_push_repository_for_job_token_allowed. This allows same-project job tokens
to push with their initiating user's repository permissions; it applies to
the project's jobs, not only this publisher. No new long-lived token is needed.
Keep branch/tag protections and cross-project pushes unchanged. The main merge
must be performed by a Maintainer allowed to create the protected release tag.

The publisher automatically saves release-qualification.json with artifact
identity, the GitLab publication job URL, actual SDK JUnit summary and linked
@@ -31,8 +39,11 @@ workflow is needed.
If a required job fails, fix/retry it or use an explicitly approved targeted CI
change through the normal review process.

For upload failures, retry the same job with the original artifacts. Inspect the
registry first if a partial upload occurred; never replace published bytes.
For upload failures, retry the same job with the original artifacts. The publisher
checks registry hashes, skips identical files and uploads only missing files.
A different checksum fails. A tag at another commit is never moved or replaced;
later main work at that already-tagged version skips publication. A new source
release needs a new package version.
A source, version or rebuilt-artifact change needs a fresh version/tag and checks.

## Order across repositories
+109 −5
Changes for ci/release.py: 109 added lines, 5 removed lines.
Original line number Diff line number Diff line
@@ -8,6 +8,8 @@ import subprocess
import sys
import zipfile
from email.parser import BytesParser
from urllib.parse import urlencode
from urllib.request import Request, urlopen

from packaging.version import Version
from reports import collect_reports
@@ -23,6 +25,85 @@ def check_tag(tag, version, protected):
        raise ValueError("Publication requires a protected release tag")


def release_context(version):
    tag = os.getenv("CI_COMMIT_TAG", "")
    if tag:
        return tag, False
    if (
        os.getenv("CI_COMMIT_BRANCH") != "main"
        or os.getenv("CI_COMMIT_REF_PROTECTED") != "true"
        or os.getenv("CI_PIPELINE_SOURCE") != "push"
    ):
        raise ValueError("Automatic publication requires a protected main push")
    if Version(version).is_prerelease:
        raise ValueError("Use a stable package version for automatic main publication")
    return f"v{version}-release", True


def ensure_main_tag(tag, revision, *, create=True):
    ref = f"refs/tags/{tag}"
    raw = subprocess.check_output(
        ["git", "ls-remote", "--tags", "origin", ref, ref + "^{}"], text=True
    )
    refs = {name: sha for sha, name in (line.split() for line in raw.splitlines())}
    existing = refs.get(ref + "^{}", refs.get(ref))
    if existing:
        return existing == revision  # A later main merge cannot move a released version.
    if not create:
        return True
    try:
        subprocess.run(
            ["git", "push", "origin", f"{revision}:{ref}"], check=True, capture_output=True
        )
    except subprocess.CalledProcessError:
        raise ValueError(
            "Tag creation failed; enable same-project job-token Git pushes "
            "and use a Maintainer-triggered pipeline"
        ) from None
    return True


def registry_json(path):
    base = f"{os.environ['CI_API_V4_URL']}/projects/{os.environ['CI_PROJECT_ID']}"
    request = Request(base + path, headers={"JOB-TOKEN": os.environ["CI_JOB_TOKEN"]})
    with urlopen(request, timeout=30) as response:
        return json.load(response)


def pending_artifacts(version):
    query = urlencode(
        {
            "package_type": "pypi",
            "package_name": "openop-tfsdk",
            "package_version": version,
            "per_page": 100,
        }
    )
    existing = {}
    for package in registry_json("/packages?" + query):
        if package["name"] != "openop-tfsdk" or package["version"] != version:
            continue
        for item in registry_json(f"/packages/{package['id']}/package_files"):
            name, checksum = item["file_name"], item["file_sha256"]
            if name in existing and existing[name] != checksum:
                raise ValueError("Registry contains conflicting artifact checksums")
            existing[name] = checksum
    pending = []
    for path in sorted(DIST.iterdir()):
        if path.name not in existing:
            pending.append(path)
        elif existing[path.name] != sha256(path):
            raise ValueError("Published artifact differs; never overwrite a released version")
    return pending


def record_publication(receipt, status):
    receipt["publication_status"] = status
    write_json(OUT / "release-check.json", receipt)
    qualification = OUT / "release-qualification.json"
    write_json(qualification, {**json.loads(qualification.read_text()), **receipt})


def verify_release():
    manifest = json.loads((OUT / "manifest.json").read_text())
    for item in manifest["artifacts"]:
@@ -35,12 +116,17 @@ def verify_release():
    if metadata["Name"] != "openop-tfsdk":
        raise ValueError("Unexpected distribution name")
    version = metadata["Version"]
    check_tag(os.getenv("CI_COMMIT_TAG", ""), version, os.getenv("CI_COMMIT_REF_PROTECTED"))
    tag, automatic = release_context(version)
    check_tag(tag, version, os.getenv("CI_COMMIT_REF_PROTECTED"))
    if manifest["source_sha"] != os.environ["CI_COMMIT_SHA"]:
        raise ValueError("Build and tag source revisions differ")
    # Tag pipelines may fetch only a detached tag; refresh the allowed ancestry refs.
    subprocess.run(["git", "fetch", "origin", "+refs/heads/*:refs/remotes/origin/*"], check=True)
    branches = ["origin/develop", "origin/main", "origin/refactor/tfsdk-architecture-alignment"]
    branches = (
        ["origin/main"]
        if automatic
        else ["origin/develop", "origin/main", "origin/refactor/tfsdk-architecture-alignment"]
    )
    if not any(
        subprocess.run(
            ["git", "merge-base", "--is-ancestor", "HEAD", ref], capture_output=True
@@ -52,16 +138,18 @@ def verify_release():
    write_json(
        OUT / "release-check.json",
        {
            "tag": os.environ["CI_COMMIT_TAG"],
            "tag": tag,
            "version": version,
            "source_sha": manifest["source_sha"],
            "wheel_sha256": sha256(wheel),
            "release_type": "candidate" if Version(version).is_prerelease else "final",
            "publication_job_url": os.getenv("CI_JOB_URL"),
            "approval_source": "protected_main_merge" if automatic else "protected_tag",
        },
    )
    receipt = json.loads((OUT / "release-check.json").read_text())
    write_json(OUT / "release-qualification.json", {**receipt, **collect_reports()})
    return receipt


def main():
@@ -69,8 +157,23 @@ def main():
    parser.add_argument("action", choices=["check", "publish"])
    args = parser.parse_args()
    os.chdir(ROOT)
    verify_release()
    receipt = verify_release()
    if args.action == "publish":
        if receipt["approval_source"] == "protected_main_merge" and not ensure_main_tag(
            receipt["tag"], receipt["source_sha"], create=False
        ):
            record_publication(receipt, "skipped_existing_release_tag")
            print("This version already has a release tag; no tag or package was replaced")
            return
        pending = pending_artifacts(receipt["version"])
        if receipt["approval_source"] == "protected_main_merge" and not ensure_main_tag(
            receipt["tag"], receipt["source_sha"]
        ):
            raise ValueError("Release tag changed during publication; no package was uploaded")
        if not pending:
            record_publication(receipt, "already_published_matching_checksums")
            print("Verified artifacts are already published")
            return
        environment = dict(
            os.environ, TWINE_USERNAME="gitlab-ci-token", TWINE_PASSWORD=os.environ["CI_JOB_TOKEN"]
        )
@@ -83,11 +186,12 @@ def main():
                "upload",
                "--repository-url",
                url,
                *[str(p) for p in DIST.iterdir()],
                *[str(p) for p in pending],
            ],
            env=environment,
            check=True,
        )
        record_publication(receipt, "published")


if __name__ == "__main__":
+122 −0
Changes for tests/ci/test_main_release.py: 122 added lines, 0 removed lines.
Original line number Diff line number Diff line
"""A reviewed main merge publishes one immutable version without another token."""

import subprocess
import sys
from pathlib import Path

import pytest

CI = Path(__file__).resolve().parents[2] / "ci"
sys.path.insert(0, str(CI))
import release


@pytest.mark.parametrize(
    "branch,protected,source,allowed",
    [
        ("main", "true", "push", True),
        ("develop", "true", "push", False),
        ("main", "false", "push", False),
        ("main", "true", "merge_request_event", False),
        ("main", "true", "web", False),
    ],
)
def test_only_a_protected_main_push_supplies_automatic_release_approval(
    monkeypatch, branch, protected, source, allowed
):
    for key, value in {
        "CI_COMMIT_TAG": "",
        "CI_COMMIT_BRANCH": branch,
        "CI_COMMIT_REF_PROTECTED": protected,
        "CI_PIPELINE_SOURCE": source,
    }.items():
        monkeypatch.setenv(key, value)
    if allowed:
        assert release.release_context("2.0.0") == ("v2.0.0-release", True)
        with pytest.raises(ValueError):
            release.release_context("2.0.0rc6")
    else:
        with pytest.raises(ValueError):
            release.release_context("2.0.0")


def test_explicit_candidate_tags_keep_the_existing_publication_path(monkeypatch):
    monkeypatch.setenv("CI_COMMIT_TAG", "v2.0.0rc6")
    assert release.release_context("2.0.0rc6") == ("v2.0.0rc6", False)


def test_automatic_tag_is_created_once_and_never_moves(tmp_path, monkeypatch):
    def git(*args, cwd=None):
        return subprocess.check_output(
            ["git", *args], cwd=cwd, stderr=subprocess.DEVNULL, text=True
        ).strip()

    remote, work = tmp_path / "remote.git", tmp_path / "work"
    git("init", "--bare", str(remote))
    git("init", "-b", "main", str(work))

    def commit(message):
        git(
            "-c",
            "user.name=Release test",
            "-c",
            "user.email=test@example.invalid",
            "commit",
            "--allow-empty",
            "-m",
            message,
            cwd=work,
        )
        return git("rev-parse", "HEAD", cwd=work)

    first = commit("Reviewed source")
    git("remote", "add", "origin", str(remote), cwd=work)
    git("push", "origin", "main", cwd=work)
    monkeypatch.chdir(work)
    tag = "v2.0.0-release"
    assert release.ensure_main_tag(tag, first, create=False)
    assert git("--git-dir", str(remote), "tag", "--list") == ""
    assert release.ensure_main_tag(tag, first)
    assert release.ensure_main_tag(tag, first)
    later = commit("Later main work")
    assert not release.ensure_main_tag(tag, later)
    assert git("--git-dir", str(remote), "rev-parse", "refs/tags/" + tag) == first
    assert git("--git-dir", str(remote), "rev-parse", "refs/heads/main") == first


@pytest.mark.parametrize("state", ["absent", "partial", "complete", "different"])
def test_registry_retry_uploads_only_missing_identical_artifacts(tmp_path, monkeypatch, state):
    dist = tmp_path / "dist"
    dist.mkdir()
    wheel, sdist = dist / "sdk.whl", dist / "sdk.tar.gz"
    wheel.write_bytes(b"tested wheel")
    sdist.write_bytes(b"tested source archive")
    monkeypatch.setattr(release, "DIST", dist)
    files = (
        []
        if state == "absent"
        else [
            {
                "file_name": wheel.name,
                "file_sha256": "wrong" if state == "different" else release.sha256(wheel),
            }
        ]
    )
    if state == "complete":
        files.append({"file_name": sdist.name, "file_sha256": release.sha256(sdist)})
    monkeypatch.setattr(
        release,
        "registry_json",
        lambda path: (
            files
            if path.endswith("package_files")
            else [{"id": 1, "name": "openop-tfsdk", "version": "2.0.0"}]
        ),
    )
    if state == "different":
        with pytest.raises(ValueError, match="never overwrite"):
            release.pending_artifacts("2.0.0")
    else:
        assert set(release.pending_artifacts("2.0.0")) == (
            {wheel, sdist} if state == "absent" else {sdist} if state == "partial" else set()
        )