Skip to content

Part 2: Policy

What Will You Do

In this part of the self-paced exercise, you will create and enforce a "policy" comprised of two OPA Gatekeeper constraints. Ensure that

  1. Container resource limits to requests does not surpass a specified ratio
  2. Only container images from authorized/approved repositories are allowed to operate on clusters

OPA Constraint Framework

Gatekeeper uses the OPA Constraint Framework to describe and enforce policy. Look there for more detailed information on their semantics and advanced usage.

Constraint

A constraint is a declaration that its author wants a system to meet a given set of requirements. Each Constraint is written with Rego, a declarative query language used by OPA to enumerate instances of data that violate the expected state of the system. All Constraints are evaluated as a Logical AND. If one Constraint is not satisfied, then the whole request is rejected.

In the example below, the constraint requires users to specify labels for k8s namespaces.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: ns-must-have-label-owner
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Namespace"]
  parameters:
    labels: ["owner"]

Constraint Template

Before defining a Constraint, you need to create a Constraint Template that allows people to declare new Constraints. Each template describes both the Rego logic that enforces the Constraint and the schema for the Constraint, which includes the schema of the CRD and the parameters that can be passed into a Constraint, much like arguments to a function.

Example:

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
        listKind: K8sRequiredLabelsList
        plural: k8srequiredlabels
        singular: k8srequiredlabels
      validation:
        # Schema for the `parameters` field
        openAPIV3Schema:
          properties:
            labels:
              type: array
              items: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels

        violation[{"msg": msg, "details": {"missing_labels": missing}}] {
          provided := {label | input.review.object.metadata.labels[label]}
          required := {label | label := input.parameters.labels[_]}
          missing := required - provided
          count(missing) > 0
          msg := sprintf("you must provide labels: %v", [missing])
        }

Step 1: Create Constraint Templates

In this step, we will create two "custom" constraint templates. The constraint templates will control the allowed repositories where images can be obtained from and sets the maximum ratio for container resource limits to requests.

Creating a Constraint Template to allow container images from approved repositories

  • Login to the Controller and select Constraint Templates under OPA Gatekeeper specification
  • Click New Template
  • Provide a name for the template (e.g. rafay-gatekeeper-allowedrepos-constraint-template)
  • Select the Upload files manually option and upload the below yaml file
  • Click Save & Exit
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: opak8sallowedrepos
  annotations:
    description: Requires container images to begin with a repo string from a specified
      list.
spec:
  crd:
    spec:
      names:
        kind: OPAK8sAllowedRepos
      validation:
        # Schema for the `parameters` field
        openAPIV3Schema:
          properties:
            repos:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sallowedrepos

        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          satisfied := [good | repo = input.parameters.repos[_] ; good = re_match(repo, container.image)]
          not any(satisfied)
          msg := sprintf("container <%v> has an invalid image repo <%v>, allowed repos are %v", [container.name, container.image, input.parameters.repos])
        }

        violation[{"msg": msg}] {
          container := input.review.object.spec.initContainers[_]
          satisfied := [good | repo = input.parameters.repos[_] ; good = re_match(repo, container.image)]
          not any(satisfied)
          msg := sprintf("container <%v> has an invalid image repo <%v>, allowed repos are %v", [container.name, container.image, input.parameters.repos])
        }

Creating a Constraint Template to check for container resource limits to requests ratio

  • Login to the Controller and select Constraint Templates under OPA Gatekeeper specification
  • Click New Template
  • Provide a name for the template (e.g. rafay-request-limit-ratio-constraint-template)
  • Select the Upload files manually option and upload the below yaml file
  • Click Save & Exit
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: opak8scontainerratios
  annotations:
    description: Sets a maximum ratio for container resource limits to requests.
spec:
  crd:
    spec:
      names:
        kind: OPAK8sContainerRatios
      validation:
        openAPIV3Schema:
          properties:
            ratio:
              type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8scontainerratios

        missing(obj, field) = true {
          not obj[field]
        }

        missing(obj, field) = true {
          obj[field] == ""
        }

        canonify_cpu(orig) = new {
          is_number(orig)
          new := orig * 1000
        }

        canonify_cpu(orig) = new {
          not is_number(orig)
          endswith(orig, "m")
          new := to_number(replace(orig, "m", ""))
        }

        canonify_cpu(orig) = new {
          not is_number(orig)
          not endswith(orig, "m")
          re_match("^[0-9]+$", orig)
          new := to_number(orig) * 1000
        }

        canonify_cpu(orig) = new {
          not is_number(orig)
          not endswith(orig, "m")
          re_match("^[0-9]+[.][0-9]+$", orig)
          new := to_number(orig) * 1000
        }

        # 10 ** 21
        mem_multiple("E") = 1000000000000000000000 { true }

        # 10 ** 18
        mem_multiple("P") = 1000000000000000000 { true }

        # 10 ** 15
        mem_multiple("T") = 1000000000000000 { true }

        # 10 ** 12
        mem_multiple("G") = 1000000000000 { true }

        # 10 ** 9
        mem_multiple("M") = 1000000000 { true }

        # 10 ** 6
        mem_multiple("k") = 1000000 { true }

        # 10 ** 3
        mem_multiple("") = 1000 { true }

        # Kubernetes accepts millibyte precision when it probably shouldn't.
        # https://github.com/kubernetes/kubernetes/issues/28741
        # 10 ** 0
        mem_multiple("m") = 1 { true }

        # 1000 * 2 ** 10
        mem_multiple("Ki") = 1024000 { true }

        # 1000 * 2 ** 20
        mem_multiple("Mi") = 1048576000 { true }

        # 1000 * 2 ** 30
        mem_multiple("Gi") = 1073741824000 { true }

        # 1000 * 2 ** 40
        mem_multiple("Ti") = 1099511627776000 { true }

        # 1000 * 2 ** 50
        mem_multiple("Pi") = 1125899906842624000 { true }

        # 1000 * 2 ** 60
        mem_multiple("Ei") = 1152921504606846976000 { true }

        get_suffix(mem) = suffix {
          not is_string(mem)
          suffix := ""
        }

        get_suffix(mem) = suffix {
          is_string(mem)
          count(mem) > 0
          suffix := substring(mem, count(mem) - 1, -1)
          mem_multiple(suffix)
        }

        get_suffix(mem) = suffix {
          is_string(mem)
          count(mem) > 1
          suffix := substring(mem, count(mem) - 2, -1)
          mem_multiple(suffix)
        }

        get_suffix(mem) = suffix {
          is_string(mem)
          count(mem) > 1
          not mem_multiple(substring(mem, count(mem) - 1, -1))
          not mem_multiple(substring(mem, count(mem) - 2, -1))
          suffix := ""
        }

        get_suffix(mem) = suffix {
          is_string(mem)
          count(mem) == 1
          not mem_multiple(substring(mem, count(mem) - 1, -1))
          suffix := ""
        }

        get_suffix(mem) = suffix {
          is_string(mem)
          count(mem) == 0
          suffix := ""
        }

        canonify_mem(orig) = new {
          is_number(orig)
          new := orig * 1000
        }

        canonify_mem(orig) = new {
          not is_number(orig)
          suffix := get_suffix(orig)
          raw := replace(orig, suffix, "")
          re_match("^[0-9]+$", raw)
          new := to_number(raw) * mem_multiple(suffix)
        }

        violation[{"msg": msg}] {
          general_violation[{"msg": msg, "field": "containers"}]
        }

        violation[{"msg": msg}] {
          general_violation[{"msg": msg, "field": "initContainers"}]
        }

        general_violation[{"msg": msg, "field": field}] {
          container := input.review.object.spec[field][_]
          cpu_orig := container.resources.limits.cpu
          not canonify_cpu(cpu_orig)
          msg := sprintf("container <%v> cpu limit <%v> could not be parsed", [container.name, cpu_orig])
        }

        general_violation[{"msg": msg, "field": field}] {
          container := input.review.object.spec[field][_]
          mem_orig := container.resources.limits.memory
          not canonify_mem(mem_orig)
          msg := sprintf("container <%v> memory limit <%v> could not be parsed", [container.name, mem_orig])
        }

        general_violation[{"msg": msg, "field": field}] {
          container := input.review.object.spec[field][_]
          cpu_orig := container.resources.requests.cpu
          not canonify_cpu(cpu_orig)
          msg := sprintf("container <%v> cpu request <%v> could not be parsed", [container.name, cpu_orig])
        }

        general_violation[{"msg": msg, "field": field}] {
          container := input.review.object.spec[field][_]
          mem_orig := container.resources.requests.memory
          not canonify_mem(mem_orig)
          msg := sprintf("container <%v> memory request <%v> could not be parsed", [container.name, mem_orig])
        }

        general_violation[{"msg": msg, "field": field}] {
          container := input.review.object.spec[field][_]
          not container.resources
          msg := sprintf("container <%v> has no resource limits", [container.name])
        }

        general_violation[{"msg": msg, "field": field}] {
          container := input.review.object.spec[field][_]
          not container.resources.limits
          msg := sprintf("container <%v> has no resource limits", [container.name])
        }

        general_violation[{"msg": msg, "field": field}] {
          container := input.review.object.spec[field][_]
          missing(container.resources.limits, "cpu")
          msg := sprintf("container <%v> has no cpu limit", [container.name])
        }

        general_violation[{"msg": msg, "field": field}] {
          container := input.review.object.spec[field][_]
          missing(container.resources.limits, "memory")
          msg := sprintf("container <%v> has no memory limit", [container.name])
        }

        general_violation[{"msg": msg, "field": field}] {
          container := input.review.object.spec[field][_]
          not container.resources.requests
          msg := sprintf("container <%v> has no resource requests", [container.name])
        }

        general_violation[{"msg": msg, "field": field}] {
          container := input.review.object.spec[field][_]
          missing(container.resources.requests, "cpu")
          msg := sprintf("container <%v> has no cpu request", [container.name])
        }

        general_violation[{"msg": msg, "field": field}] {
          container := input.review.object.spec[field][_]
          missing(container.resources.requests, "memory")
          msg := sprintf("container <%v> has no memory request", [container.name])
        }

        general_violation[{"msg": msg, "field": field}] {
          container := input.review.object.spec[field][_]
          cpu_limits_orig := container.resources.limits.cpu
          cpu_limits := canonify_cpu(cpu_limits_orig)
          cpu_requests_orig := container.resources.requests.cpu
          cpu_requests := canonify_cpu(cpu_requests_orig)
          cpu_ratio := input.parameters.ratio
          to_number(cpu_limits) > to_number(cpu_ratio) * to_number(cpu_requests)
          msg := sprintf("container <%v> cpu limit <%v> is higher than the maximum allowed ratio of <%v>", [container.name, cpu_limits_orig, cpu_ratio])
        }

        general_violation[{"msg": msg, "field": field}] {
          container := input.review.object.spec[field][_]
          mem_limits_orig := container.resources.limits.memory
          mem_requests_orig := container.resources.requests.memory
          mem_limits := canonify_mem(mem_limits_orig)
          mem_requests := canonify_mem(mem_requests_orig)
          mem_ratio := input.parameters.ratio
          to_number(mem_limits) > to_number(mem_ratio) * to_number(mem_requests)
          msg := sprintf("container <%v> memory limit <%v> is higher than the maximum allowed ratio of <%v>", [container.name, mem_limits_orig, mem_ratio])
        }

Constraint Templates

Step 2: Create Constraints

In this step, we will create two constraints that are associated with the previously created constraint templates.

Creating a Constraint to allow container images from approved repositories

  • Login to the Controller and select Constraints under OPA Gatekeeper specification
  • Click New Constraint
  • Provide a name for the constraint (e.g. rafay-gatekeeper-allowedrepos-constraint) and select the Constraint Template (rafay-gatekeeper-allowedrepos-constraint-template)
  • Specify the version name (e.g. v1)
  • Select the Upload files manually option and upload the below yaml file
  • Click Save Changes
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: OPAK8sAllowedRepos
metadata:
  name: opa-allowed-repos
spec:
  enforcementAction: deny
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    repos:
      - "amazonaws.com" # ECR registry for EKS cluster
      - "k8s.gcr.io" # Kubernetes registry
      - "docker.io" # bitnami registry

Creating a Constraint to check for container resource limits to requests ratio

  • Login to the Controller and select Constraints under OPA Gatekeeper specification
  • Click New Constraint
  • Provide a name for the constraint (e.g. rafay-request-limit-ratio-constraint) and select the Constraint Template (rafay-request-limit-ratio-constraint-template)
  • Specify the version name (e.g. v1)
  • Select the Upload files manually option and upload the below yaml file
  • Click Save Changes
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: OPAK8sContainerRatios
metadata:
  name: opa-container-must-meet-ratio
spec:
  enforcementAction: warn
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    ratio: "1"

Constraints

Step 3: Create Policy

In this step, we will create a policy which will assemble together multiple constraints that were created in the prior steps.

  • Login to the Controller and select Policies under OPA Gatekeeper section
  • Click New Policy, provide a name (e.g. opa-gs-policy) and a version (e.g. v1)
  • Click Add Constraint and select the constraints previously created along with their versions
  • Click Save Changes

Policy

Recap

Congratulations! At this point, you have successfully created a policy with two custom constraints. In the next step, you will add this policy to a cluster blueprint and deploy it on a fleet of managed clusters.

Step 1: Create Constraint Templates

In this step, we will create two "custom" constraint templates. The constraint templates will control the allowed repositories where images can be obtained from and sets the maximum ratio for container resource limits to requests.

  • Open Terminal (on macOS/Linux) or Command Prompt (Windows) and navigate to the folder where you forked the Git repository
  • Navigate to the folder "/getstarted/opa_gatekeeper/constraint_templates"

The "constraint-template-1.yaml" and "constraint-template-2.yaml" files contain the declarative specifications for our two constraint templates. These specification files point to the constraint template definitions located in the "rafay-gatekeeper-allowedrepos-constraint-template.yaml" and "rafay-request-limit-ratio-constraint-template.yaml" files.

The below YAML shows the declarative specification details found in the "constraint-template-1.yaml" file.

Important

Ensure you update the "project: defaultproject" with the name of the project in your Org

apiVersion: opa.k8smgmt.io/v3
kind: OPAConstraintTemplate
metadata:
  labels:
    rafay.dev/opa: template
  name: rafay-gatekeeper-allowedrepos-constraint-template
  project: defaultproject
spec:
  artifact:
    artifact:
      paths:
      - name: file://rafay-gatekeeper-allowedrepos-constraint-template.yaml
    options: {}
    type: Yaml

The below YAML shows the constraint template definition found in the "rafay-gatekeeper-allowedrepos-constraint-template.yaml" file

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: opak8sallowedrepos
  annotations:
    description: Requires container images to begin with a repo string from a specified
      list.
spec:
  crd:
    spec:
      names:
        kind: OPAK8sAllowedRepos
      validation:
        # Schema for the `parameters` field
        openAPIV3Schema:
          properties:
            repos:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sallowedrepos

        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          satisfied := [good | repo = input.parameters.repos[_] ; good = re_match(repo, container.image)]
          not any(satisfied)
          msg := sprintf("container <%v> has an invalid image repo <%v>, allowed repos are %v", [container.name, container.image, input.parameters.repos])
        }

        violation[{"msg": msg}] {
          container := input.review.object.spec.initContainers[_]
          satisfied := [good | repo = input.parameters.repos[_] ; good = re_match(repo, container.image)]
          not any(satisfied)
          msg := sprintf("container <%v> has an invalid image repo <%v>, allowed repos are %v", [container.name, container.image, input.parameters.repos])
        }
  • Type the commands below to create the constraint templates
rctl apply -f constraint-template-1.yaml
rctl apply -f constraint-template-2.yaml

If you did not encounter any errors, you can optionally verify if everything was created correctly on the controller.

  • Navigate to the "defaultproject" project in your Org
  • Select OPA Gatekepper -> Constraint Templates

Constraint Templates

Step 2: Create Constraints

In this step, we will create two constraints that are assocaited with the previously created constraint templates.

  • Open Terminal (on macOS/Linux) or Command Prompt (Windows) and navigate to the folder where you forked the Git repository
  • Navigate to the folder "/getstarted/opa_gatekeeper/constraints"

The "constraint-1.yaml" and "constraint-2.yaml" files contain the declarative specification for our two constraints. These specification files point to the constraint definitions located in the "rafay-gatekeeper-allowedrepos-constraint.yaml" and "rafay-request-limit-ratio-constraint.yaml" files.

The below YAML shows the declarative specification details found in the "constraint-1.yaml" file

Important

Ensure you update the "project: defaultproject" with the name of the project in your Org

apiVersion: opa.k8smgmt.io/v3
kind: OPAConstraint
metadata:
  labels:
    rafay.dev/opa: constraint
  name: rafay-gatekeeper-allowedrepos-constraint
  project: defaultproject
spec:
  artifact:
    artifact:
      paths:
      - name: file://rafay-gatekeeper-allowedrepos-constraint.yaml
    options: {}
    type: Yaml
  published: true
  templateName: rafay-gatekeeper-allowedrepos-constraint-template
  version: v1

The below YAML shows the constraint template definition found in the "rafay-gatekeeper-allowedrepos-constraint.yaml" file

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: OPAK8sAllowedRepos
metadata:
  name: opa-allowed-repos
spec:
  enforcementAction: deny
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    repos:
      - "amazonaws.com" # ECR registry for EKS cluster
      - "k8s.gcr.io" # Kubernetes registry
      - "docker.io" # bitnami registry
  • Type the commands below to create the constraints
rctl apply -f constraint-1.yaml
rctl apply -f constraint-2.yaml

If you did not encounter any errors, you can optionally verify if everything was created correctly on the controller.

  • Navigate to the "defaultproject" project in your Org
  • Select OPA Gatekepper -> Constraints

Constraints

Step 3: Create Policy

In this step, we will create a policy which will assemble together multiple constraints that were created in the prior steps.

  • Open Terminal (on macOS/Linux) or Command Prompt (Windows) and navigate to the folder where you forked the Git repository
  • Navigate to the folder "/getstarted/opa_gatekeeper/policy"

The "policy.yaml" file contains the declarative specification for the policy. This specification file refers to the previously created constraints as part of the defintition of the policy.

The below YAML shows the declarative specification details found in the "policy.yaml" file

Important

Ensure you update the "project: defaultproject" with the name of the project in your Org

apiVersion: opa.k8smgmt.io/v3
kind: OPAPolicy
metadata:
  name: opa-gs-policy
  project: defaultproject
spec:
  constraintList:
  - name: rafay-request-limit-ratio-constraint
    version: v1
  - name: rafay-gatekeeper-allowedrepos-constraint
    version: v1
  sharing:
    enabled: true
    projects:
    - name: defaultproject
  version: opa-gs-policy-version
  • Type the command below to create the policy
rctl apply -f policy.yaml

If you did not encounter any errors, you can optionally verify if everything was created correctly on the controller.

  • Navigate to the "defaultproject" project in your Org
  • Select OPA Gatekeeper -> Policies

Policy

Recap

Congratulations! At this point, you have successfully created a policy with two custom constraints. In the next step, you will add this policy to a cluster blueprint and deploy it on a fleet of managed clusters.