test: the scheduler image on the OCM hub (!1) · Merge requests · OSL / MDG HypO / code / addons / ocm-scheduler

chart/ocm-scheduler-chart/crds/hyposchedulingmanifests.scheduling.hypo.eu.yaml

+4 −5

Changes for chart/ocm-scheduler-chart/crds/hyposchedulingmanifests.scheduling.hypo.eu.yaml: 4 added lines, 5 removed lines.
Original line number	Diff line number	Diff line
		@@ -2,7 +2,7 @@ apiVersion: apiextensions.k8s.io/v1
		kind: CustomResourceDefinition
		metadata:
		annotations:
		controller-gen.kubebuilder.io/version: v0.16.1
		controller-gen.kubebuilder.io/version: v0.19.0
		name: hyposchedulingmanifests.scheduling.hypo.eu
		spec:
		group: scheduling.hypo.eu
		@@ -41,8 +41,7 @@ spec:
		metadata:
		type: object
		spec:
		description: HypoSchedulingManifestSpec defines the desired state of
		HypoSchedulingManifest
		description: HypoSchedulingManifestSpec defines the desired state of HypoSchedulingManifest
		properties:
		globalAnnotations:
		description: \|-
		@@ -86,8 +85,8 @@ spec:
		- manifests
		type: object
		status:
		description: HypoSchedulingManifestStatus defines the observed state
		of HypoSchedulingManifest
		description: HypoSchedulingManifestStatus defines the observed state of
		HypoSchedulingManifest
		properties:
		conditions:
		description: List of conditions for HypoSchedulingManifest

chart/README.md

+36 −8

Changes for chart/README.md: 36 added lines, 8 removed lines.
Original line number	Diff line number	Diff line
		@@ -100,11 +100,11 @@ clusterSet:
		managedClusters:
		- name: managed-1
		location: managed-1
		kubeconfigSecretName: managed1-kubeconfig
		kubeconfigSecretId: <hypo-registry-kubeconfig-id>

		- name: managed-2
		location: managed-2
		kubeconfigSecretName: managed2-kubeconfig
		kubeconfigSecretId: <hypo-registry-kubeconfig-id>
		```

		### Scheduler configuration
		@@ -128,9 +128,9 @@ managedClusters:

		\| Parameter \| Description \|
		\|---\|---\|
		\| `managedClusters[].name` \| Name of the ManagedCluster registered on the OCM Hub. \|
		\| `managedClusters[].name` \| Name of the ManagedCluster registered on the OCM Hub. Also the name of the Kubernetes Secret holding its kubeconfig. \|
		\| `managedClusters[].location` \| Location value assigned to the cluster through the `hypo.filter.location` ClusterClaim. \|
		\| `managedClusters[].kubeconfigSecretName` \| Name of the Kubernetes Secret containing the managed cluster kubeconfig. \|
		\| `managedClusters[].kubeconfigSecretId` \| HypO registry ID of the managed cluster kubeconfig. When deployed from HypO, the package manager fetches it and creates the Secret `<name>` before install. Not used by the chart templates. \|

		The `location` value is used by the scheduler when evaluating location-based scheduling requirements. It should match the location expected by the corresponding workload scheduling configuration.

		@@ -138,16 +138,20 @@ The `location` value is used by the scheduler when evaluating location-based sch

		The Helm setup and cleanup Jobs require access to the managed clusters to create and remove ClusterClaims.

		If we deploy manually the Hlem chart and not from HypO, we need for each managed cluster, to create a Kubernetes Secret in the scheduler namespace containing its kubeconfig.
		When deployed from HypO, the package manager creates these Secrets from `managedClusters[].kubeconfigSecretId` before installing the chart.

		If we deploy manually the Helm chart and not from HypO, we need for each managed cluster, to create a Kubernetes Secret in the scheduler namespace containing its kubeconfig, named after the cluster's `name`.

		This is the case for the OCM hub used by this repository's CI (see [The OCM hub used by the CI](#the-ocm-hub-used-by-the-ci)). The Secrets must stay in place there, since the setup Job reads them again on every `helm upgrade`.

		Example:

		```bash
		kubectl create secret generic managed1-kubeconfig \
		kubectl create secret generic managed-1 \
		--from-file=kubeconfig=/home/ubuntu/managed1.yaml \
		-n hypo-scheduler-system

		kubectl create secret generic managed2-kubeconfig \
		kubectl create secret generic managed-2 \
		--from-file=kubeconfig=/home/ubuntu/managed2.yaml \
		-n hypo-scheduler-system
		```
		@@ -160,7 +164,7 @@ Verify the Secrets:
		kubectl get secrets -n hypo-scheduler-system
		```

		The Secret names must match the `kubeconfigSecretName` values in `values.yaml`.
		The Secret names must match the `managedClusters[].name` values in `values.yaml`.

		### Private container registry

		@@ -316,6 +320,30 @@ The Helm post-upgrade Job reapplies the managed cluster configuration, including

		Verify the updated configuration using the checks in Section 7.

		### The OCM hub used by the CI

		The OCM hub and this chart (release `ocm-scheduler` in `hypo-scheduler-system`) are installed there by hand, once, before the first pipeline run. The hub is also used by the nightly tests.

		The merge request pipeline only tests the Go code of the scheduler:

		1. It builds the image `hypo-ocm-scheduler:<commit>`.
		2. It sets it on the Deployment `ocm-scheduler-controller-manager` with `kubectl set image`. Helm is not used, and the chart, CRD and RBAC on the hub stay unchanged.
		3. It runs the HypO OCM scenario (`@smoke-e2e-workflow-ocm`).
		4. If the tests pass, the new image stays on the hub. If they fail, the previous image is put back.

		Changes to the Helm chart, the CRD (`chart/ocm-scheduler-chart/crds/`) or the RBAC are applied to the hub by hand after the merge. Since the pipeline does not record the image in the Helm release, pass the image currently running on the hub, or Helm puts back the tag stored in the release (run from `chart/`):

		```bash
		TAG=$(kubectl get deploy ocm-scheduler-controller-manager -n hypo-scheduler-system \
		-o jsonpath="{.spec.template.spec.containers[?(@.name=='manager')].image}" \| sed 's/.*://')

		kubectl apply --server-side --force-conflicts -f ocm-scheduler-chart/crds/ # only if the CRD changed
		helm upgrade ocm-scheduler ./ocm-scheduler-chart -n hypo-scheduler-system --reuse-values \
		--set scheduler.image.tag=$TAG --wait --wait-for-jobs --timeout 10m
		```

		Helm never updates the files in `crds/` on `helm upgrade`, hence the `kubectl apply`.

		## 9. Uninstall the Scheduler

		The chart includes a Helm `pre-delete` cleanup Job that removes scheduler-specific resources before Helm deletes the release.

src/ocm_scheduler_operator/config/crd/bases/scheduling.hypo.eu_hyposchedulingmanifests.yaml

+4 −5

Changes for src/ocm_scheduler_operator/config/crd/bases/scheduling.hypo.eu_hyposchedulingmanifests.yaml: 4 added lines, 5 removed lines.
Original line number	Diff line number	Diff line
		@@ -3,7 +3,7 @@ apiVersion: apiextensions.k8s.io/v1
		kind: CustomResourceDefinition
		metadata:
		annotations:
		controller-gen.kubebuilder.io/version: v0.16.1
		controller-gen.kubebuilder.io/version: v0.19.0
		name: hyposchedulingmanifests.scheduling.hypo.eu
		spec:
		group: scheduling.hypo.eu
		@@ -42,8 +42,7 @@ spec:
		metadata:
		type: object
		spec:
		description: HypoSchedulingManifestSpec defines the desired state of
		HypoSchedulingManifest
		description: HypoSchedulingManifestSpec defines the desired state of HypoSchedulingManifest
		properties:
		globalAnnotations:
		description: \|-
		@@ -87,8 +86,8 @@ spec:
		- manifests
		type: object
		status:
		description: HypoSchedulingManifestStatus defines the observed state
		of HypoSchedulingManifest
		description: HypoSchedulingManifestStatus defines the observed state of
		HypoSchedulingManifest
		properties:
		conditions:
		description: List of conditions for HypoSchedulingManifest

src/ocm_scheduler_operator/deploy/hypo-ocm-scheduler.yaml

+1 −1

Changes for src/ocm_scheduler_operator/deploy/hypo-ocm-scheduler.yaml: 1 added line, 1 removed line.
Original line number	Diff line number	Diff line
		@@ -11,7 +11,7 @@ apiVersion: apiextensions.k8s.io/v1
		kind: CustomResourceDefinition
		metadata:
		annotations:
		controller-gen.kubebuilder.io/version: v0.16.1
		controller-gen.kubebuilder.io/version: v0.19.0
		name: hyposchedulingmanifests.scheduling.hypo.eu
		spec:
		group: scheduling.hypo.eu

src/ocm_scheduler_operator/internal/controller/hyposchedulingmanifest_controller_test.go

+90 −34

Changes for src/ocm_scheduler_operator/internal/controller/hyposchedulingmanifest_controller_test.go: 90 added lines, 34 removed lines.
Original line number	Diff line number	Diff line
		@@ -21,64 +21,120 @@ import (

		. "github.com/onsi/ginkgo/v2"
		. "github.com/onsi/gomega"
		corev1 "k8s.io/api/core/v1"
		"k8s.io/apimachinery/pkg/api/errors"
		"k8s.io/apimachinery/pkg/api/meta"
		metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
		"k8s.io/apimachinery/pkg/runtime"
		"k8s.io/apimachinery/pkg/types"
		"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
		"sigs.k8s.io/controller-runtime/pkg/reconcile"

		metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

		schedulingv1alpha1 "github.com/rh-waterford-et/hypo-scheduler-operator/api/v1alpha1"
		)

		// envtest has no OCM CRDs, so these specs cover the reconcile paths that return before any
		// ManagedClusterSet, Placement or ManifestWork lookup.
		var _ = Describe("HypoSchedulingManifest Controller", func() {
		Context("When reconciling a resource", func() {
		const resourceName = "test-resource"
		const (
		resourceName = "test-resource"
		schedulerNs = "hypo-scheduler-system"
		otherNs = "default"
		deploymentObject = `{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"name":"web"},` +
		`"spec":{"selector":{"matchLabels":{"app":"web"}},"template":{"metadata":{"labels":{"app":"web"}},` +
		`"spec":{"containers":[{"name":"web","image":"nginx"}]}}}}`
		)

		ctx := context.Background()
		var reconciler *HypoSchedulingManifestReconciler
		var previousSchedulerNs string

		typeNamespacedName := types.NamespacedName{
		Name: resourceName,
		Namespace: "default", // TODO(user):Modify as needed
		}
		hyposchedulingmanifest := &schedulingv1alpha1.HypoSchedulingManifest{}

		BeforeEach(func() {
		By("creating the custom resource for the Kind HypoSchedulingManifest")
		err := k8sClient.Get(ctx, typeNamespacedName, hyposchedulingmanifest)
		if err != nil && errors.IsNotFound(err) {
		createManifest := func(namespace string, globalAnnotations []string) types.NamespacedName {
		key := types.NamespacedName{Name: resourceName, Namespace: namespace}
		resource := &schedulingv1alpha1.HypoSchedulingManifest{
		ObjectMeta: metav1.ObjectMeta{
		Name: resourceName,
		Namespace: "default",
		ObjectMeta: metav1.ObjectMeta{Name: key.Name, Namespace: key.Namespace},
		Spec: schedulingv1alpha1.HypoSchedulingManifestSpec{
		GlobalAnnotations: globalAnnotations,
		Manifests: []runtime.RawExtension{{Raw: []byte(deploymentObject)}},
		},
		// TODO(user): Specify other spec details if needed.
		}
		Expect(k8sClient.Create(ctx, resource)).To(Succeed())
		return key
		}
		})

		AfterEach(func() {
		// TODO(user): Cleanup logic after each test, like removing the resource instance.
		reconcileAndGet := func(key types.NamespacedName) *schedulingv1alpha1.HypoSchedulingManifest {
		_, err := reconciler.Reconcile(ctx, reconcile.Request{NamespacedName: key})
		Expect(err).NotTo(HaveOccurred())

		resource := &schedulingv1alpha1.HypoSchedulingManifest{}
		err := k8sClient.Get(ctx, typeNamespacedName, resource)
		Expect(k8sClient.Get(ctx, key, resource)).To(Succeed())
		return resource
		}

		expectCondition := func(resource *schedulingv1alpha1.HypoSchedulingManifest, conditionType, reason string) {
		condition := meta.FindStatusCondition(resource.Status.Conditions, conditionType)
		Expect(condition).NotTo(BeNil())
		Expect(condition.Status).To(Equal(metav1.ConditionTrue))
		Expect(condition.Reason).To(Equal(reason))
		Expect(resource.Status.State).To(Equal(conditionType))
		}

		BeforeEach(func() {
		previousSchedulerNs = HypoSchedulerNamespace
		HypoSchedulerNamespace = schedulerNs

		ns := &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: schedulerNs}}
		if err := k8sClient.Create(ctx, ns); err != nil && !errors.IsAlreadyExists(err) {
		Expect(err).NotTo(HaveOccurred())
		}

		By("Cleanup the specific resource instance HypoSchedulingManifest")
		Expect(k8sClient.Delete(ctx, resource)).To(Succeed())
		reconciler = &HypoSchedulingManifestReconciler{Client: k8sClient, Scheme: k8sClient.Scheme()}
		})
		It("should successfully reconcile the resource", func() {
		By("Reconciling the created resource")
		controllerReconciler := &HypoSchedulingManifestReconciler{
		Client: k8sClient,
		Scheme: k8sClient.Scheme(),

		AfterEach(func() {
		HypoSchedulerNamespace = previousSchedulerNs

		for _, namespace := range []string{schedulerNs, otherNs} {
		key := types.NamespacedName{Name: resourceName, Namespace: namespace}
		resource := &schedulingv1alpha1.HypoSchedulingManifest{}
		if err := k8sClient.Get(ctx, key, resource); errors.IsNotFound(err) {
		continue
		}
		// The finalizer's cleanup needs OCM, so it is removed directly
		if controllerutil.RemoveFinalizer(resource, finalizer) {
		Expect(k8sClient.Update(ctx, resource)).To(Succeed())
		}
		Expect(k8sClient.Delete(ctx, resource)).To(Succeed())
		}
		})

		It("ignores a resource outside the scheduler namespace", func() {
		key := createManifest(otherNs, []string{"hypo.target.managedClusterSet=global"})

		_, err := controllerReconciler.Reconcile(ctx, reconcile.Request{
		NamespacedName: typeNamespacedName,
		resource := reconcileAndGet(key)

		expectCondition(resource, misconfigured, "IncorrectNamespace")
		Expect(controllerutil.ContainsFinalizer(resource, finalizer)).To(BeFalse())
		})
		Expect(err).NotTo(HaveOccurred())
		// TODO(user): Add more specific assertions depending on your controller's reconciliation logic.
		// Example: If you expect a certain status condition after reconciliation, verify it here.

		It("rejects an unsupported annotation", func() {
		key := createManifest(schedulerNs, []string{
		"hypo.target.managedClusterSet=global",
		"hypo.unknown=value",
		})

		resource := reconcileAndGet(key)

		expectCondition(resource, misconfigured, "UnsupportedAnnotation")
		Expect(controllerutil.ContainsFinalizer(resource, finalizer)).To(BeTrue())
		})

		It("requires a target managed cluster set", func() {
		key := createManifest(schedulerNs, []string{"hypo.filter.location=greece"})

		resource := reconcileAndGet(key)

		expectCondition(resource, misconfigured, "MissingTarget")
		Expect(controllerutil.ContainsFinalizer(resource, finalizer)).To(BeTrue())
		})
		})

test: the scheduler image on the OCM hub

Merge request reports