Security - preventing container escape on a shared Kubernetes / Kubeflow cluster

1. General problem overview

On a shared Kubernetes cluster, a Kubeflow user may be able to create Pods directly or indirectly through:

Kubeflow Pipelines,
Notebooks,
KServe,
Deployments,
Jobs.

If the cluster allows a Pod with dangerous settings such as:

securityContext:
  privileged: true

hostPID: true

or:

volumes:
  - name: host-root
    hostPath:
      path: /

the container may gain access to resources of the Kubernetes node on which it is running.

The most dangerous combination is:

privileged: true,
hostPID: true,
a hostPath volume pointing to /.

Such a Pod may allow a user to escape normal container isolation and access the node operating system.

The purpose of this test is to verify that Kubernetes rejects such Pods during admission, before they are created.

2. Important safety rules

This test may be executed in production only with:

--dry-run=server

This mode sends the manifest to the Kubernetes API and runs RBAC and admission checks, but it does not save or start the Pod.

Do not:

remove --dry-run=server,
run kubectl exec,
run nsenter,
attempt to access the node,
schedule the test Pod on a specific node.

If the result contains:

created (server dry run)

Kubernetes accepted the manifest, and the Pod would be created if --dry-run=server were removed.

3. Set the namespace

Set the namespace in which the tests will be executed:

export NS="<your_namespace>"

Every tester must replace test2 with the correct Kubeflow user namespace.

Verify the value:

echo "$NS"

Expected result:

<your_namespace>

4. Minimal test set

The minimal test set consists of:

Checking that the kubeconfig is not administrative.
Checking whether the user can create Pods.
Running one fast decisive test.
Running three detailed tests.
Running one safe control Pod test.

4.1. Check the current identity

kubectl auth whoami

Example valid result for a regular user:

Username    system:serviceaccount:test2:edit-token
Groups      [system:serviceaccounts system:serviceaccounts:test2 system:authenticated]

Interpretation:

The user should not be a cluster administrator.
An edit-token account or a regular Kubeflow user is suitable for the test.
Testing only with an administrator account does not confirm the behavior for a regular Kubeflow user.

4.2. Check for full administrative permissions

kubectl auth can-i '*' '*' --all-namespaces

Expected result:

no

If the result is:

yes

stop the test and switch to a regular user kubeconfig.

Do not continue with an administrative kubeconfig because the result will not represent the permissions of a normal Kubeflow user.

4.3. Check whether the user can create Pods

kubectl auth can-i create pods -n "$NS"

Possible results:

yes: The user can create Pods directly. Admission controls must therefore block dangerous Pod settings.
no: The user cannot create Pods directly. The user may still be able to create them indirectly through Pipeline, Notebook, KServe, Deployment, or Job resources. In that case, a direct Pod test may fail because of RBAC, but indirect workload paths still need to be checked.

5. Fast decisive test

This test combines the three main vectors:

privileged: true,
hostPID: true,
hostPath: /.

kubectl create --dry-run=server -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: container-escape-combined-test
  namespace: ${NS}
  annotations:
    sidecar.istio.io/inject: "false"
spec:
  restartPolicy: Never
  hostPID: true
  containers:
    - name: test
      image: busybox:1.36
      command: ["sleep", "300"]
      securityContext:
        privileged: true
      volumeMounts:
        - name: host-root
          mountPath: /host
  volumes:
    - name: host-root
      hostPath:
        path: /
        type: Directory
EOF

Expected secure result

Error from server (Forbidden)

The message may mention:

privileged,
hostPID,
hostPath,
PodSecurity,
Kyverno,
a baseline or restricted policy violation.

Insecure result

pod/container-escape-combined-test created (server dry run)

At this point, the security test has already failed.

This means that at least one path exists for creating a Pod that may access the node.

The test may be stopped and the issue reported. However, it is recommended to run the three detailed tests below to identify exactly which controls are missing.

6. Detailed tests

6.1. `privileged: true` only

kubectl create --dry-run=server -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: privileged-test
  namespace: ${NS}
  annotations:
    sidecar.istio.io/inject: "false"
spec:
  containers:
    - name: test
      image: busybox:1.36
      command: ["sleep", "300"]
      securityContext:
        privileged: true
EOF

Expected secure result:

Error from server (Forbidden)

Insecure result:

pod/privileged-test created (server dry run)

If the Pod is accepted, the test fails. Further tests are optional but useful for determining the full scope of the issue.

6.2. `hostPID: true` only

kubectl create --dry-run=server -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: hostpid-test
  namespace: ${NS}
  annotations:
    sidecar.istio.io/inject: "false"
spec:
  hostPID: true
  containers:
    - name: test
      image: busybox:1.36
      command: ["sleep", "300"]
EOF

Expected secure result:

Error from server (Forbidden)

Insecure result:

pod/hostpid-test created (server dry run)

If the Pod is accepted, the user can start a container that can see node processes.

6.3. `hostPath: /` only

kubectl create --dry-run=server -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: hostpath-test
  namespace: ${NS}
  annotations:
    sidecar.istio.io/inject: "false"
spec:
  containers:
    - name: test
      image: busybox:1.36
      command: ["sleep", "300"]
      volumeMounts:
        - name: host-root
          mountPath: /host
  volumes:
    - name: host-root
      hostPath:
        path: /
        type: Directory
EOF

Expected secure result:

Error from server (Forbidden)

Insecure result:

pod/hostpath-test created (server dry run)

If the Pod is accepted, the user can mount the node filesystem inside the container.

7. Safe control Pod

This test verifies that the policy does not block all Pods and only rejects dangerous configurations.

kubectl create --dry-run=server -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: safe-control-test
  namespace: ${NS}
  annotations:
    sidecar.istio.io/inject: "false"
spec:
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: test
      image: busybox:1.36
      command: ["sleep", "300"]
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
EOF

Expected result:

pod/safe-control-test created (server dry run)

If the safe Pod also returns Forbidden, inspect the error message. The cause may be another policy, a missing required field, or a cluster-specific restriction.

8. Pass criteria

The test passes only when:

Test	Expected result
Kubeconfig is not administrative	`no` for full permissions
Pod with `privileged: true`	`Forbidden`
Pod with `hostPID: true`	`Forbidden`
Pod with `hostPath: /`	`Forbidden`
Pod combining all three settings	`Forbidden`
Safe control Pod	`created (server dry run)`

9. When the test may be stopped early

If any dangerous Pod returns:

created (server dry run)

the security test has already failed.

The test may be stopped and the issue reported.

It is still useful to run the remaining tests because they show whether the cluster fails to block:

privileged mode,
the host PID namespace,
hostPath volumes.

10. Optional extended tests

These tests are not required for minimal confirmation, but they help assess the wider security scope.

10.1. Check whether other workload types can be created

kubectl auth can-i create deployments.apps -n "$NS"

kubectl auth can-i create jobs.batch -n "$NS"

kubectl auth can-i create workflows.argoproj.io -n "$NS"

kubectl auth can-i create notebooks.kubeflow.org -n "$NS"

kubectl auth can-i create inferenceservices.serving.kserve.io -n "$NS"

A yes result means the user may be able to create Pods indirectly through a controller.

10.2. Check whether the user can modify the namespace

kubectl auth can-i patch "namespace/$NS"

kubectl auth can-i update "namespace/$NS"

Expected result for a regular user:

no

10.3. Check Pod Security Admission labels

kubectl get namespace "$NS" --show-labels

Look for:

pod-security.kubernetes.io/enforce=baseline

or:

pod-security.kubernetes.io/enforce=restricted

Missing labels do not automatically mean that no protection exists. The cluster may use Kyverno, Gatekeeper, or another admission webhook. The final source of truth is the --dry-run=server result.

11. Expected result after remediation

After remediation:

privileged: true must be blocked,
hostPID: true must be blocked,
hostPath must be blocked or tightly restricted,
safe Pods should still be accepted,
the protection should also apply to Pods created through Kubeflow Pipelines, Notebooks, and KServe.

The protection may be implemented using:

Pod Security Admission with baseline or restricted,
Kyverno policies running in Enforce mode,
ValidatingAdmissionPolicy,
Gatekeeper.

Security - preventing container escape on a shared Kubernetes / Kubeflow cluster

1. General problem overview

2. Important safety rules

3. Set the namespace

4. Minimal test set

4.1. Check the current identity

4.2. Check for full administrative permissions

4.3. Check whether the user can create Pods

5. Fast decisive test

Expected secure result

Insecure result

6. Detailed tests

6.1. privileged: true only

6.2. hostPID: true only

6.3. hostPath: / only

7. Safe control Pod

8. Pass criteria

9. When the test may be stopped early

10. Optional extended tests

10.1. Check whether other workload types can be created

10.2. Check whether the user can modify the namespace

10.3. Check Pod Security Admission labels

11. Expected result after remediation

6.1. `privileged: true` only

6.2. `hostPID: true` only

6.3. `hostPath: /` only