Terminal and kubectl Katib validation
Scenario 1. Successfully launch a Katib experiment from the terminal
This scenario verifies that a Katib experiment can be started successfully from a local terminal by using the Katib client.
Prerequisites
Before starting, make sure that you have:
a functioning MLOps Studio environment,
access to a local terminal with Python 3.x,
a working kubectl connection to the cluster.
Steps
Launch a virtual environment and install the Katib client.
python3 -m venv katib-env source katib-env/bin/activate pip install kubeflow-katib
Check if the user’s namespace has the correct label.
kubectl get namespace <namespace_name> --show-labels
The labels must include:
katib.kubeflow.org/metrics-collector-injection=enabled
Start port-forward to the Katib Controller in a separate terminal.
kubectl port-forward -n kubeflow svc/katib-controller 8081:443
Note
This terminal must remain open throughout the duration of the experiment.
Prepare the katib_test.py file with the experiment configuration.
Get the file from the Katib website:
https://www.kubeflow.org/docs/components/katib/getting-started/
Important
If you want to run it in your namespace, you must change the following in this Python file:
katib_client = katib.KatibClient(namespace="kubeflow")
to:
katib_client = katib.KatibClient(namespace="<namespace_name>")
Run the experiment.
python katib_test.pyKatib will create test pods (trials) in your namespace.
Monitor the experiment progress.
To see the trials:
kubectl get pods -n <namespace_name> | grep tune-experiment
Example of a correct state:
tune-experiment-xxxxx 2/2 Running tune-experiment-yyyyy 2/2 Running ...
Results
After the experiment is completed:
kubectl get experiments -n <namespace_name>
shows the state:
Succeeded
All trials in the namespace transitioned to the Completed state.
The Python script prints the best found hyperparameters, for example:
Optimal trial: a=17, b=0.13
A new experiment with appropriate data assigned in the respective tabs should appear in MLOps Studio → Katib.
Scenario 2. Katib experiment with parallel trials and resource limits defined in YAML and launched via kubectl apply
This scenario verifies that a Katib experiment can run multiple trials in parallel while using resource requests and limits defined in YAML.
Steps
Create a katib-parallel-resources.yaml file with the following content:
apiVersion: kubeflow.org/v1beta1 kind: Experiment metadata: namespace: <namespace_name> name: parallel-resources-test spec: objective: type: minimize goal: 0.01 objectiveMetricName: loss algorithm: algorithmName: random parallelTrialCount: 2 maxTrialCount: 4 maxFailedTrialCount: 2 parameters: - name: lr parameterType: double feasibleSpace: min: "0.01" max: "0.05" - name: momentum parameterType: double feasibleSpace: min: "0.5" max: "0.9" trialTemplate: primaryContainerName: training-container trialParameters: - name: learningRate description: Learning rate reference: lr - name: momentum description: SGD momentum reference: momentum trialSpec: apiVersion: batch/v1 kind: Job spec: template: metadata: annotations: sidecar.istio.io/inject: "false" spec: containers: - name: training-container image: ghcr.io/kubeflow/katib/pytorch-mnist-cpu:latest command: - "python3" - "/opt/pytorch-mnist/mnist.py" - "--epochs=1" - "--batch-size=16" - "--lr=${trialParameters.learningRate}" - "--momentum=${trialParameters.momentum}" resources: requests: cpu: "0.25" memory: "256Mi" limits: cpu: "0.5" memory: "512Mi" restartPolicy: Never
Apply the Katib experiment manifest using kubectl.
kubectl apply -f katib-parallel-resources.yaml
Verify that the experiment has been created and has a Running or Created status.
kubectl get experiments -n <namespace_name>
Wait for the experiment to finish and verify its final status. It should be Succeeded. This may take up to a dozen or so minutes.
kubectl get experiments -n <namespace_name>
Additionally, you can inspect the detailed status.
kubectl describe experiment parallel-resources-test -n <namespace_name>
Results
The parallel-resources-test experiment was created in your namespace based on the katib-parallel-resources.yaml file.
Katib launched a maximum of 2 trials in parallel according to the parallelTrialCount setting, until maxTrialCount was reached.
Each trial had requests and limits for CPU/Memory set as defined in the trialTemplate.
After all trials were completed, the experiment transitioned to the Succeeded state.
All trial pods finished correctly (Completed), without resource errors or startup issues.
If the UI is configured correctly, the experiment is visible in MLOps Studio → Katib Experiments after selecting the correct namespace/profile.
Scenario 3. Lack of available resources
This scenario verifies how Katib behaves when the requested resources exceed what the cluster can currently schedule.
Steps
Create a katib-overload-resources.yaml file.
apiVersion: kubeflow.org/v1beta1 kind: Experiment metadata: namespace: <namespace_name> name: resource-overload-test spec: objective: type: minimize goal: 0.01 objectiveMetricName: loss algorithm: algorithmName: random parallelTrialCount: 10 maxTrialCount: 20 maxFailedTrialCount: 10 parameters: - name: lr parameterType: double feasibleSpace: min: "0.01" max: "0.1" - name: momentum parameterType: double feasibleSpace: min: "0.5" max: "0.9" trialTemplate: primaryContainerName: training-container trialParameters: - name: learningRate description: Learning rate reference: lr - name: momentum description: SGD momentum reference: momentum trialSpec: apiVersion: batch/v1 kind: Job spec: template: metadata: annotations: sidecar.istio.io/inject: "false" spec: containers: - name: training-container image: ghcr.io/kubeflow/katib/pytorch-mnist-cpu:latest command: - "python3" - "/opt/pytorch-mnist/mnist.py" - "--epochs=3" - "--batch-size=64" - "--lr=${trialParameters.learningRate}" - "--momentum=${trialParameters.momentum}" resources: requests: cpu: "4" memory: "8Gi" limits: cpu: "4" memory: "8Gi" restartPolicy: Never
Run via:
kubectl apply -f katib-overload-resources.yaml
Check the pod status.
kubectl describe pod <pod_name> -n <namespace_name>
Results
The experiment will not start, and the error will be of the type:
Warning FailedScheduling 3m6s default-scheduler 0/15 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 14 Insufficient cpu, 14 Insufficient memory. preemption: 0/15 nodes are available: 1 Preemption is not helpful for scheduling, 14 No preemption victims found for incoming pod.Trial pods are stuck in Pending, unable to transition en masse to Running.
kubectl get pods -n <namespace_name>