KServe

This article describes the main KServe deployment methods and outlines their operational characteristics in a MLOps Studio environment. It also explains the differences between the Raw Deployment and Serverless approaches, and shows how metrics can be integrated into the MLOps Studio Central Dashboard.

Overview

KServe supports different approaches to running inference workloads in Kubernetes. In this context, the two main methods are Raw Deployment and Serverless.

The choice between them affects infrastructure overhead, exposure of the inference endpoint outside the cluster, scaling behavior, and the level of integration with the MLOps Studio Central Dashboard.

Prerequisites

Before starting, make sure that:

  1. You have access to a MLOps Studio environment with KServe enabled.

  2. You can open the KServe UI in the MLOps Studio Central Dashboard.

  3. Your environment includes the components required for the deployment method that you want to use.

  4. For the Serverless method, knative-serving is installed in addition to KServe.

  5. If you want the inference to be exposed outside the cluster, your environment supports the required networking configuration.

Raw Deployment method

The method assumes running inference in standard Kubernetes cluster deployments. It reduces infrastructure overhead due to the smaller number of components needed for operation, as it requires only KServe.

The fundamental downside of this method is that in a MLOps Studio cluster integrated with Istio, it does not automatically expose the inference outside the cluster. In a MLOps Studio cluster, this requires creating a VirtualService object, whereas only an Ingress-type object is created. This is an intentional design choice by the KServe creators, and addressing it requires the end user to manually create a VirtualService, which in turn requires knowledge of Istio and its definitions.

For reference, see the related upstream issue:

Additionally, custom integration with the MLOps Studio Central Dashboard would be required, as it is designed to work only with inferences created in Serverless mode.

Serverless method

The method assumes running inference using the knative-serving component. It carries a higher infrastructure overhead, requiring the installation of knative-serving in addition to KServe, but does not require additional configuration from the end user. Simply adding the inference is sufficient, and all remaining configuration is performed automatically, including exposing the inference outside the cluster.

An additional benefit is the scaling of inference based on usage. Knative enables scaling to zero when there is no demand, which reduces costs associated with model serving. Furthermore, the Serverless method automatically integrates with the MLOps Studio Central Dashboard, allowing insight into model details directly from the web UI.

Metrics

By utilizing the Serverless method, it is possible to integrate metrics with the MLOps Studio Central Dashboard, enabling the user to monitor the model without the need to manually build a monitoring environment. Such functionality in the Raw Deployment method would require building monitoring from scratch, including custom dashboards and integration with the MLOps Studio Central Dashboard.

../../../../_images/kserve-endpoint-metrics.png

The Metrics view in MLOps Studio can then be used to inspect resource usage directly for the deployed endpoint.

What to do next

After reviewing the KServe deployment methods, the next useful topics are:

  • validating KServe behavior and deployment outcomes in more detail,

  • checking how KServe interacts with Istio resources such as VirtualService objects,

  • comparing operational trade-offs between simpler and more automated serving approaches,

  • preparing reusable KServe manifests for supported model formats.