Terminal and kubectl Katib validation

Scenario 1. Successfully launch a Katib experiment from the terminal

This scenario verifies that a Katib experiment can be started successfully from a local terminal by using the Katib client.

Prerequisites

Before starting, make sure that you have:

  • a functioning MLOps Studio environment,

  • access to a local terminal with Python 3.x,

  • a working kubectl connection to the cluster.

Steps

  1. Launch a virtual environment and install the Katib client.

    python3 -m venv katib-env
    source katib-env/bin/activate
    pip install kubeflow-katib
    
  2. Check if the user’s namespace has the correct label.

    kubectl get namespace <namespace_name> --show-labels
    

    The labels must include:

    katib.kubeflow.org/metrics-collector-injection=enabled
    
  3. Start port-forward to the Katib Controller in a separate terminal.

    kubectl port-forward -n kubeflow svc/katib-controller 8081:443
    

    Note

    This terminal must remain open throughout the duration of the experiment.

  4. Prepare the katib_test.py file with the experiment configuration.

    Get the file from the Katib website:

    https://www.kubeflow.org/docs/components/katib/getting-started/

    Important

    If you want to run it in your namespace, you must change the following in this Python file:

    katib_client = katib.KatibClient(namespace="kubeflow")
    

    to:

    katib_client = katib.KatibClient(namespace="<namespace_name>")
    
  5. Run the experiment.

    python katib_test.py
    

    Katib will create test pods (trials) in your namespace.

  6. Monitor the experiment progress.

    To see the trials:

    kubectl get pods -n <namespace_name> | grep tune-experiment
    

    Example of a correct state:

    tune-experiment-xxxxx   2/2  Running
    tune-experiment-yyyyy   2/2  Running
    ...
    

Results

  1. After the experiment is completed:

    kubectl get experiments -n <namespace_name>
    

    shows the state:

    Succeeded
    
  2. All trials in the namespace transitioned to the Completed state.

  3. The Python script prints the best found hyperparameters, for example:

    Optimal trial:
    a=17, b=0.13
    
  4. A new experiment with appropriate data assigned in the respective tabs should appear in MLOps Studio → Katib.

Scenario 2. Katib experiment with parallel trials and resource limits defined in YAML and launched via kubectl apply

This scenario verifies that a Katib experiment can run multiple trials in parallel while using resource requests and limits defined in YAML.

Steps

  1. Create a katib-parallel-resources.yaml file with the following content:

    apiVersion: kubeflow.org/v1beta1
    kind: Experiment
    metadata:
      namespace: <namespace_name>
      name: parallel-resources-test
    spec:
      objective:
        type: minimize
        goal: 0.01
        objectiveMetricName: loss
      algorithm:
        algorithmName: random
    
      parallelTrialCount: 2
      maxTrialCount: 4
      maxFailedTrialCount: 2
    
      parameters:
        - name: lr
          parameterType: double
          feasibleSpace:
            min: "0.01"
            max: "0.05"
        - name: momentum
          parameterType: double
          feasibleSpace:
            min: "0.5"
            max: "0.9"
    
      trialTemplate:
        primaryContainerName: training-container
        trialParameters:
          - name: learningRate
            description: Learning rate
            reference: lr
          - name: momentum
            description: SGD momentum
            reference: momentum
        trialSpec:
          apiVersion: batch/v1
          kind: Job
          spec:
            template:
              metadata:
                annotations:
                  sidecar.istio.io/inject: "false"
              spec:
                containers:
                  - name: training-container
                    image: ghcr.io/kubeflow/katib/pytorch-mnist-cpu:latest
                    command:
                      - "python3"
                      - "/opt/pytorch-mnist/mnist.py"
                      - "--epochs=1"
                      - "--batch-size=16"
                      - "--lr=${trialParameters.learningRate}"
                      - "--momentum=${trialParameters.momentum}"
                    resources:
                      requests:
                        cpu: "0.25"
                        memory: "256Mi"
                      limits:
                        cpu: "0.5"
                        memory: "512Mi"
                restartPolicy: Never
    
  2. Apply the Katib experiment manifest using kubectl.

    kubectl apply -f katib-parallel-resources.yaml
    
  3. Verify that the experiment has been created and has a Running or Created status.

    kubectl get experiments -n <namespace_name>
    
  4. Wait for the experiment to finish and verify its final status. It should be Succeeded. This may take up to a dozen or so minutes.

    kubectl get experiments -n <namespace_name>
    
  5. Additionally, you can inspect the detailed status.

    kubectl describe experiment parallel-resources-test -n <namespace_name>
    

Results

  1. The parallel-resources-test experiment was created in your namespace based on the katib-parallel-resources.yaml file.

  2. Katib launched a maximum of 2 trials in parallel according to the parallelTrialCount setting, until maxTrialCount was reached.

  3. Each trial had requests and limits for CPU/Memory set as defined in the trialTemplate.

  4. After all trials were completed, the experiment transitioned to the Succeeded state.

  5. All trial pods finished correctly (Completed), without resource errors or startup issues.

  6. If the UI is configured correctly, the experiment is visible in MLOps Studio → Katib Experiments after selecting the correct namespace/profile.

Scenario 3. Lack of available resources

This scenario verifies how Katib behaves when the requested resources exceed what the cluster can currently schedule.

Steps

  1. Create a katib-overload-resources.yaml file.

    apiVersion: kubeflow.org/v1beta1
    kind: Experiment
    metadata:
      namespace: <namespace_name>
      name: resource-overload-test
    spec:
      objective:
        type: minimize
        goal: 0.01
        objectiveMetricName: loss
      algorithm:
        algorithmName: random
    
      parallelTrialCount: 10
      maxTrialCount: 20
      maxFailedTrialCount: 10
    
      parameters:
        - name: lr
          parameterType: double
          feasibleSpace:
            min: "0.01"
            max: "0.1"
        - name: momentum
          parameterType: double
          feasibleSpace:
            min: "0.5"
            max: "0.9"
    
      trialTemplate:
        primaryContainerName: training-container
        trialParameters:
          - name: learningRate
            description: Learning rate
            reference: lr
          - name: momentum
            description: SGD momentum
            reference: momentum
        trialSpec:
          apiVersion: batch/v1
          kind: Job
          spec:
            template:
              metadata:
                annotations:
                  sidecar.istio.io/inject: "false"
              spec:
                containers:
                  - name: training-container
                    image: ghcr.io/kubeflow/katib/pytorch-mnist-cpu:latest
                    command:
                      - "python3"
                      - "/opt/pytorch-mnist/mnist.py"
                      - "--epochs=3"
                      - "--batch-size=64"
                      - "--lr=${trialParameters.learningRate}"
                      - "--momentum=${trialParameters.momentum}"
                    resources:
                      requests:
                        cpu: "4"
                        memory: "8Gi"
                      limits:
                        cpu: "4"
                        memory: "8Gi"
                restartPolicy: Never
    
  2. Run via:

    kubectl apply -f katib-overload-resources.yaml
    
  3. Check the pod status.

    kubectl describe pod <pod_name> -n <namespace_name>
    

Results

  1. The experiment will not start, and the error will be of the type:

    Warning  FailedScheduling   3m6s   default-scheduler   0/15 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 14 Insufficient cpu, 14 Insufficient memory. preemption: 0/15 nodes are available: 1 Preemption is not helpful for scheduling, 14 No preemption victims found for incoming pod.
    
  2. Trial pods are stuck in Pending, unable to transition en masse to Running.

    kubectl get pods -n <namespace_name>