Policies

What Will You Do

In this exercise,

• You will be defining certain policies and make sure that your resources in the cluster are adhering to those policies.

---

Assumptions

• You have already provisioned or imported a Kubernetes cluster using the controller.
• You have successfully published a Gatekeeper addon based cluster blueprint to your cluster.

---

OPA Constraint Framework

Gatekeeper uses the OPA Constraint Framework to describe and enforce policy. Look there for more detailed information on their semantics and advanced usage.

Constraint

A constraint is a declaration that its author wants a system to meet a given set of requirements.

Exaxmple:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: ns-must-have-label-owner
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Namespace"]
  parameters:
    labels: ["owner"]

---

Constraint Template

Constraint Templates allow people to declare new constraints. They can provide the expected input parameters and the underlying Rego necessary to enforce their intent.

Example:

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
        listKind: K8sRequiredLabelsList
        plural: k8srequiredlabels
        singular: k8srequiredlabels
      validation:
        # Schema for the `parameters` field
        openAPIV3Schema:
          properties:
            labels:
              type: array
              items: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels

        violation[{"msg": msg, "details": {"missing_labels": missing}}] {
          provided := {label | input.review.object.metadata.labels[label]}
          required := {label | label := input.parameters.labels[_]}
          missing := required - provided
          count(missing) > 0
          msg := sprintf("you must provide labels: %v", [missing])
        }

---

Applying the policy to the cluster

• Copy the constraint template above to a file called labels-policy.yaml.
• Append the contraint from above to the same file (labels-policy.yaml). Make sure both constraint template and constraint are delimited by "---" in the yaml file.
• Login into the Web Console and navigate to your Project as an Org Admin or Infrastructure Admin
• Create a workload (type: k8s yaml), upload the labels-policy.yaml file from the previous step,
• Select a cluster and publish it

Once the workload is published, you should be able to see the constraint template and constraint in the cluster.

kubectl get constrainttemplate
NAME                       AGE
k8srequiredlabels          57s

kubectl get constraint
NAME                                                                   AGE
k8srequiredlabels.constraints.gatekeeper.sh/ns-must-have-label-owner   76s

---

Verifying the policy

Let's try to create the namespace.

kubectl create ns demo
Error from server ([denied by ns-must-have-label-owner] you must provide labels: {"owner"}): admission webhook "validation.gatekeeper.sh" denied the request: [denied by ns-must-have-label-owner] you must provide labels: {"owner"}

OPA Gatekeeper did not allow us to create the namespace as it did not have the owner label.

Now let's create the namespace with the owner label.

cat << EOF | kubectl create -f -
apiVersion: v1
kind: Namespace
metadata:
  name: demo
  labels:
    "owner": "demo.example.com"
EOF
namespace/demo created

OPA Gatekeeper Audit

The audit functionality enables periodic evaluations of replicated resources against the policies enforced in the cluster to detect pre-existing misconfigurations. Audit results are stored as violations listed in the status field of the failed constraint.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  annotations:
    rafay.dev/original: '{"kind":"K8sRequiredLabels","spec":{"match":{"kinds":[{"kinds":["Namespace"],"apiGroups":[""]}]},"parameters":{"labels":["owner"]}},"metadata":{"name":"ns-must-have-label-owner","labels":{"rep-partner":"rx28oml","rep-project":"z24wnmy","rep-workload":"ns-labels-opa","rep-organization":"5m18rky"},"namespace":"gatekeeper-system"},"apiVersion":"constraints.gatekeeper.sh/v1beta1"}'
    rafay.dev/ownerRef: '{"apiVersion":"cluster.rafay.dev/v2","kind":"Tasklet","name":"ns-labels-opa","uid":"ea8e3a3d-d553-435f-a218-4ff566c55e48","controller":true,"blockOwnerDeletion":true}'
  creationTimestamp: "2020-08-25T23:31:32Z"
  generation: 1
  labels:
    rep-organization: 5m18rky
    rep-partner: rx28oml
    rep-project: z24wnmy
    rep-workload: ns-labels-opa
  name: ns-must-have-label-owner
  resourceVersion: "1181250"
  selfLink: /apis/constraints.gatekeeper.sh/v1beta1/k8srequiredlabels/ns-must-have-label-owner
  uid: 177c89cb-f654-431c-ab76-99df9fa8e8ab
spec:
  match:
    kinds:
    - apiGroups:
      - ""
      kinds:
      - Namespace
  parameters:
    labels:
    - owner
status:
  auditTimestamp: "2020-08-25T23:41:06Z"
  byPod:
  - constraintUID: 177c89cb-f654-431c-ab76-99df9fa8e8ab
    enforced: true
    id: gatekeeper-audit-6d9bf5d549-qnl54
    observedGeneration: 1
    operations:
    - audit
    - status
  - constraintUID: 177c89cb-f654-431c-ab76-99df9fa8e8ab
    enforced: true
    id: gatekeeper-controller-manager-8549fb4f48-5vhvp
    observedGeneration: 1
    operations:
    - webhook
  - constraintUID: 177c89cb-f654-431c-ab76-99df9fa8e8ab
    enforced: true
    id: gatekeeper-controller-manager-8549fb4f48-7b6t2
    observedGeneration: 1
    operations:
    - webhook
  - constraintUID: 177c89cb-f654-431c-ab76-99df9fa8e8ab
    enforced: true
    id: gatekeeper-controller-manager-8549fb4f48-qf48l
    observedGeneration: 1
    operations:
    - webhook
  totalViolations: 10
  violations:
  - enforcementAction: deny
    kind: Namespace
    message: 'you must provide labels: {"owner"}'
    name: default
  - enforcementAction: deny
    kind: Namespace
    message: 'you must provide labels: {"owner"}'
    name: gatekeeper-system
  - enforcementAction: deny
    kind: Namespace
    message: 'you must provide labels: {"owner"}'
    name: kube-node-lease
  - enforcementAction: deny
    kind: Namespace
    message: 'you must provide labels: {"owner"}'
    name: kube-public
  - enforcementAction: deny
    kind: Namespace
    message: 'you must provide labels: {"owner"}'
    name: kube-system
  - enforcementAction: deny
    kind: Namespace
    message: 'you must provide labels: {"owner"}'
    name: rafay-infra
  - enforcementAction: deny
    kind: Namespace
    message: 'you must provide labels: {"owner"}'
    name: rafay-system
  - enforcementAction: deny
    kind: Namespace
    message: 'you must provide labels: {"owner"}'
    name: test
  - enforcementAction: deny
    kind: Namespace
    message: 'you must provide labels: {"owner"}'
    name: velero
  - enforcementAction: deny
    kind: Namespace
    message: 'you must provide labels: {"owner"}'
    name: wordpress

As you can see the policy that we applied earlier detected that there are 10 violations in the cluster. OPA Gatekeeper will continue to report them as violation until we fix them or exclude them from the policy.

---

Exempting Namespaces from Evaluation

There are two ways you can exempt a namespace from evaluation.

• Can be done globally for all the policies
• Can be done at a constraint level for a specific policy

Global exemption

To exempt certain namespaces globally, we need to define a Config resource which is shown as an example below:

apiVersion: config.gatekeeper.sh/v1alpha1
kind: Config
metadata:
  name: config
  namespace: "gatekeeper-system"
spec:
  match:
    - excludedNamespaces: ["default","gatekeeper-system","kube-system","rafay-infra","rafay-system"]
      processes: ["*"]

Once the above config object is applied to the cluster, default, gatekeeper-system, kube-system, rafay-infra and rafay-system namespaces will be exempted from all the policies in the cluster.

Note

valid input for processes are "audit", "webhook", "sync", "*"

Exempting a namespace for a specific policy

As an example, we want to exempt kube-system namespace for the above policy that we applied. To acheive this, we need to add excludeNamespaces to the constraint spec.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: ns-must-have-label-owner
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Namespace"]
    excludedNamespaces:
    - kube-system
  parameters:
    labels: ["owner"]

---

Dry Run

When rolling out new constraints to running clusters, the dry run functionality can be helpful as it enables constraints to be deployed in the cluster without making actual changes. This allows constraints to be tested in a running cluster without enforcing them. Cluster resources that are impacted by the dry run constraint are surfaced as violations in the status field of the constraint. To use the dry run feature, we need to add enforcementAction: dryrun to the constraint spec. By default, enforcementAction is set to deny as the default behavior is to deny admission requests with any violation.

Example:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: ns-must-have-label-owner
spec:
  enforcementAction: dryrun
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Namespace"]
  parameters:
    labels: ["owner"]

---

Data Replication

Some constraints are impossible to write without access to more state than just the object under test. For example, it is impossible to know if an ingress's hostname is unique among all ingresses unless a rule has access to all other ingresses. To make such rules possible, we need to enable syncing of data into OPA. Kubernetes data can be replicated into OPA via the sync config resource.

Example:

apiVersion: config.gatekeeper.sh/v1alpha1
kind: Config
metadata:
  name: config
  namespace: "gatekeeper-system"
spec:
  sync:
    syncOnly:
      - group: ""
        version: "v1"
        kind: "Namespace"
      - group: ""
        version: "v1"
        kind: "Pod"
      - group: "extensions"
        version: "v1beta1"
        kind: "Ingress"
      - group: "networking.k8s.io"
        version: "v1beta1"
        kind: "Ingress"

After applying the above resource to the cluster, resources for pods, namespaces, ingress data will be synced into OPA.

---

Emergency Recovery

If for some reason Gatekeeper is preventing the cluster from operating correctly, the webhook can be disabled.

kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io gatekeeper-validating-webhook-configuration

---

Examples

More example policies can be found below.

Purpose	Sample Policy Spec
Do not allow containers without any limits specified	Policy Spec
Do not allow containers without probes defined	Policy Spec
Allow containers to pull images only from ECR Registry	Policy Spec
Do not allow same ingress hostname	Policy Spec
Do not allow same service selector	Policy Spec
Allow conatiners to run with specific users	Policy Spec

---

Important

When you are importing a cluster where OPA is running already, make sure to exclude "rafay-system" and "rafay-infra" namespace from the OPA policies.

---

Recap

Congratulations! You deployed OPA policies to the cluster successfully.