Tag Archives: Security-as-Code

Enforcing Kubernetes Security at the Gate: OPA/Gatekeeper + Kyverno in Production

DevSecOps, Linux, Security, , , , , , , ,

Kubernetes RBAC is not enough. RBAC controls who can make API calls, but it does not control what those API calls can deploy. A developer with create pods permission in their namespace can deploy a container running as root, mounting the host filesystem, pulling from an untrusted registry, with no resource limits – and RBAC will not stop any of it.

This is the gap that Kubernetes Admission Controllers fill. Having hardened EKS clusters for ad-tech workloads at Smaato and energy trading platforms at work, I have learned that admission controllers are the most operationally impactful Kubernetes security control available. This post documents the production configuration I use.

How Admission Controllers Work

When a request hits the Kubernetes API server, it passes through a pipeline before being persisted to etcd:

The two relevant webhook types are:

  • Mutating Admission Webhooks: Intercept the request before validation and can modify the object. Use Kyverno here to inject secure defaults (non-root user, resource limits, labels) automatically, so developers don’t need to remember security configuration.
  • Validating Admission Webhooks: Intercept the request after mutation and either allow or deny it. Use OPA/Gatekeeper here to enforce hard policies (no privileged containers, approved registries only, required labels).

The split is intentional: Kyverno mutates to help developers, Gatekeeper validates to enforce compliance.

Installing OPA/Gatekeeper

Gatekeeper is the production-grade OPA integration for Kubernetes. Install via Helm:

helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts
helm repo update

helm install gatekeeper gatekeeper/gatekeeper \
  --namespace gatekeeper-system \
  --create-namespace \
  --set replicas=3 \
  --set auditInterval=60 \
  --set constraintViolationsLimit=100 \
  --set logLevel=WARNING

The auditInterval=60 setting is important: Gatekeeper continuously audits existing resources against all policies, not just new requests. This catches drift from resources created before the policies were installed.

Installing Kyverno

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update

helm install kyverno kyverno/kyverno \
  --namespace kyverno \
  --create-namespace \
  --set replicaCount=3 \
  --set config.webhooks[0].failurePolicy=Fail

Setting failurePolicy=Fail means if the Kyverno webhook is unavailable, API requests fail closed (denied) rather than open (allowed). This is the safer default for production.

Kyverno Mutating Policies

Policy 1: Inject Secure Container Defaults

This policy automatically injects security context into every new pod that does not already have it defined. Developers do not need to write this – Kyverno adds it transparently:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: inject-security-context
  annotations:
    policies.kyverno.io/title: Inject Secure Defaults
    policies.kyverno.io/category: Security
    policies.kyverno.io/description: >
      Injects runAsNonRoot, readOnlyRootFilesystem, and
      allowPrivilegeEscalation=false into all containers.
spec:
  rules:
    - name: inject-security-context
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: ["!kube-system", "!gatekeeper-system", "!kyverno"]
      mutate:
        patchStrategicMerge:
          spec:
            containers:
              - (name): "*"
                securityContext:
                  +(runAsNonRoot): true
                  +(readOnlyRootFilesystem): true
                  +(allowPrivilegeEscalation): false
                  +(runAsUser): 1000
            initContainers:
              - (name): "*"
                securityContext:
                  +(runAsNonRoot): true
                  +(allowPrivilegeEscalation): false

The +() syntax is Kyverno’s “add if not present” operator – it will not overwrite explicitly set values.

Policy 2: Inject Resource Limits

Pods without resource limits are a denial-of-service vector. This policy injects sensible defaults so the cluster scheduler always has resource information:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-resource-limits
spec:
  rules:
    - name: add-default-resource-limits
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: ["!kube-system"]
      mutate:
        patchStrategicMerge:
          spec:
            containers:
              - (name): "*"
                resources:
                  +(requests):
                    memory: "64Mi"
                    cpu: "50m"
                  +(limits):
                    memory: "512Mi"
                    cpu: "500m"

Policy 3: Add Mandatory Labels for NetworkPolicy

Network policies use label selectors. If pods don’t have consistent labels, network policies become fragile. This policy ensures every pod carries the labels required for policy enforcement:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-team-labels
spec:
  rules:
    - name: add-labels-from-namespace
      match:
        any:
          - resources:
              kinds: [Pod]
      context:
        - name: namespaceLabels
          apiCall:
            urlPath: "/api/v1/namespaces/{{request.object.metadata.namespace}}"
            jmesPath: "metadata.labels"
      mutate:
        patchStrategicMerge:
          metadata:
            labels:
              +(app.kubernetes.io/managed-by): "helm"
              +(security.rohanbhagat.com/team): "{{namespaceLabels.\"team\" || 'unknown'}}"
              +(security.rohanbhagat.com/environment): "{{namespaceLabels.\"environment\" || 'unknown'}}"

OPA/Gatekeeper Validating Policies

Gatekeeper uses ConstraintTemplates (the Rego logic) and Constraints (the parameters). Each policy is a pair.

Policy 1: Block Privileged Containers

Privileged containers have full access to the host kernel. This policy denies any pod spec that requests privileged mode, host network, or host PID:

# constraint-template: no-privileged-containers.yaml
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8snoPrivilegedContainers
spec:
  crd:
    spec:
      names:
        kind: K8sNoPrivilegedContainers
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8snoprivilegedcontainers

        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          container.securityContext.privileged == true
          msg := sprintf("Container '%v' must not run as privileged", [container.name])
        }

        violation[{"msg": msg}] {
          input.review.object.spec.hostPID == true
          msg := "Pod must not use hostPID"
        }

        violation[{"msg": msg}] {
          input.review.object.spec.hostNetwork == true
          msg := "Pod must not use hostNetwork"
        }

        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          container.securityContext.capabilities.add[_] == "NET_ADMIN"
          msg := sprintf("Container '%v' may not add NET_ADMIN capability", [container.name])
        }

# constraint: no-privileged-containers.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sNoPrivilegedContainers
metadata:
  name: no-privileged-containers
spec:
  enforcementAction: deny
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    excludedNamespaces:
      - kube-system
      - gatekeeper-system
      - kyverno

Policy 2: Approved Container Registries Only

Supply chain attacks start with untrusted images. This policy denies any image not from the approved ECR registry or the internal registry:

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8sapprovedregistries
spec:
  crd:
    spec:
      names:
        kind: K8sApprovedRegistries
      validation:
        openAPIV3Schema:
          type: object
          properties:
            allowedRegistries:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sapprovedregistries

        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          not image_from_approved_registry(container.image)
          msg := sprintf(
            "Container '%v' uses unapproved image '%v'. Use one of: %v",
            [container.name, container.image, input.parameters.allowedRegistries]
          )
        }

        image_from_approved_registry(image) {
          registry := input.parameters.allowedRegistries[_]
          startswith(image, registry)
        }

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sApprovedRegistries
metadata:
  name: approved-registries-only
spec:
  enforcementAction: deny
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    excludedNamespaces: ["kube-system"]
  parameters:
    allowedRegistries:
      - "123456789012.dkr.ecr.eu-central-1.amazonaws.com"
      - "registry.k8s.io"
      - "quay.io/kyverno"

Policy 3: Block latest Tag

The latest tag makes deployments non-reproducible and bypasses security scanning (you scan one digest, deploy a different one). This policy enforces explicit tags or digest references:

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8snolatestimage
spec:
  crd:
    spec:
      names:
        kind: K8sNoLatestImage
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8snolatestimage

        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          endswith(container.image, ":latest")
          msg := sprintf("Container '%v' uses ':latest' tag. Use an explicit version or digest.", [container.name])
        }

        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          not contains(container.image, ":")
          msg := sprintf("Container '%v' has no tag. Specify an explicit version or SHA digest.", [container.name])
        }

Audit Mode vs Enforce Mode

Rolling out admission controllers to an existing cluster without prior audit is high-risk – you will likely break existing workloads. Use this three-phase rollout:

Phase 1 – Audit (week 1-2): Set enforcementAction: warn in all Constraints. Gatekeeper logs violations but does not block. Review the audit report to understand current posture:

kubectl get constraint -A -o json | jq '.items[].status.totalViolations'

Phase 2 – Dry-run (week 3-4): Switch to enforcementAction: dryrun. Violations appear in kubectl describe constraint but requests are still allowed. Alert on high-violation counts.

Phase 3 – Enforce (week 5+): Switch to enforcementAction: deny. Coordinate with engineering teams to fix any remaining violations beforehand.

Testing Policies with conftest

Before deploying policy changes, test them against Kubernetes manifests locally using conftest:

# Install conftest
brew install conftest

# Test a Kubernetes manifest against your OPA policies
conftest test k8s/deployment.yaml \
  --policy policies/gatekeeper/ \
  --namespace k8s

<em># Example output:</em>
<em># FAIL - k8s/deployment.yaml - Container 'app' uses ':latest' tag.</em>
<em># FAIL - k8s/deployment.yaml - Container 'app' must not run as privileged.</em>
<em># 2 tests, 0 passed, 0 warnings, 2 failures</em>

Integrate conftest into the CI/CD pipeline to catch policy violations before they reach the cluster:

# .github/workflows/k8s-policy-check.yml
- name: Validate K8s manifests against policies
  run: |
    conftest test k8s/ \
      --policy policies/gatekeeper/ \
      --namespace k8s \
      --output github

Namespace-Level Network Policies as a Complement

Admission controllers control what runs in the cluster. Network policies control how workloads communicate. The two work together. After the label-injection Kyverno policy ensures all pods have consistent labels, these Network Policies enforce zero-trust within the cluster:

# Default deny-all for every namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
# Allow intra-namespace traffic only
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-same-namespace
spec:
  podSelector: {}
  ingress:
    - from:
        - podSelector: {}
  egress:
    - to:
        - podSelector: {}
  policyTypes: [Ingress, Egress]

Results

After deploying this configuration on a 40-node EKS production cluster at Smaato:

  • Zero privileged containers running in production (down from 8 before enforcement)
  • 100% of pods have explicit resource limits (up from ~40% before mutation policies)
  • CI policy gate catches manifest violations in 90 seconds, before the image is even built
  • CIS Kubernetes Benchmark score on control plane 5.x (Admission Control) moved from 3/9 to 9/9 controls passing