MLOps Studio Pipelines: Accessing Model Registry via Istio

When building MLOps Studio Pipelines (KFP), tasks that need to communicate with internal mesh services – such as the Model Registry – require a specific Istio annotation.

Without this annotation, pipeline components may fail to resolve or connect to the Model Registry service, resulting in timeouts or connection errors. This guide explains the architectural reason behind this requirement, how to implement it, and how to troubleshoot related network failures.

Why is this annotation needed?

MLOps Studio operates on top of a service mesh (Istio) to handle internal routing, security, and traffic management. However, there is a fundamental architectural conflict between how Istio and MLOps Studio’s underlying execution engine (Argo Workflows) handle container lifecycles:

  • Argo Workflows (Ephemeral): Argo expects pipeline tasks to run to completion. A step is only marked as “Successful” when all containers inside its pod terminate gracefully.

  • Istio Sidecars (Persistent): Istio injects an Envoy proxy sidecar container into pods to route network traffic. This proxy is designed to run continuously as a background daemon and does not exit on its own.

If Istio sidecars were automatically injected into every pipeline step, the main pipeline script would finish, but the Envoy proxy would keep running. This would cause the pipeline step to hang in a Running state indefinitely, creating a deadlock.

To prevent pipeline deadlocks, automatic Istio sidecar injection is disabled by default for pipeline pods.

The Catch: Because the model-registry-service sits behind Istio networking policies, you cannot communicate with it using its standard service hostname without participating in the service mesh. While direct pod-to-pod IP access might technically bypass the mesh, communicating securely and reliably via the MLOps Studio service infrastructure requires the Envoy proxy. Therefore, you must manually “opt-in” to the sidecar injection strictly for the pipeline steps that perform these external API calls.

How to use the annotation

To enable mesh communication for a specific pipeline task, you must apply the sidecar.istio.io/inject: “true” annotation using the KFP Kubernetes extensions.

Apply this only to the components making network requests (e.g., REST API calls via requests or using the ModelRegistry Python client).

Python Example (KFP v2):

from kfp import dsl
from kfp import kubernetes

@dsl.pipeline(name="example-pipeline")
def my_pipeline():
    # 1. Define your task
    register_task = register_version(
        model_name="my-model",
        model_uri="s3://path/to/model",
    )

    # 2. Add the Istio sidecar annotation
    kubernetes.add_pod_annotation(
        register_task,
        annotation_key="sidecar.istio.io/inject",
        annotation_value="true",
    )

Troubleshooting missing Istio sidecars

If you omit the annotation, even experienced Kubernetes users might struggle to identify the root cause, as the errors often manifest as generic network failures.

Common Symptoms

  • Connection Timeouts: Python throws requests.exceptions.ConnectTimeout or urllib3.exceptions.MaxRetryError when trying to reach http://model-registry-service:8080.

  • Connection Refused: The logs show Connection refused errors, indicating the pod cannot negotiate the route to the service.

  • 503 Service Unavailable: Occasionally, if the request routes partially but fails mTLS validation without the sidecar, you may see HTTP 503 errors.

How to Verify the Issue

If a pipeline step failing with network errors is suspected to be missing its Istio proxy, inspect the pod in your Kubernetes cluster:

  1. Find the pod name associated with your failed pipeline step.

  2. Run kubectl describe pod <pod-name> -n <your-namespace>.

  3. Check the Containers section.

Diagnosis:

  • If you see only one container (the main main or wait container), the sidecar was not injected. You need to add the sidecar.istio.io/inject: “true” annotation to the task in your Python code.

  • If you see an istio-proxy container running alongside your main container, the annotation is present, and the network issue is likely caused by something else (e.g., incorrect hostnames, missing network policies, or the Model Registry service being down).

Deep Dive: Proving the Istio Proxy is the Root Cause

When a pipeline task fails with a generic network error (like a timeout or Connection refused), you can use kubectl to confirm whether the lack of an Istio sidecar is specifically responsible. KFP pods contain multiple containers (typically a main container for your code and a wait container for Argo). When Istio is injected, it adds a third container (istio-proxy).

Here is how to definitively diagnose the issue from the command line.

1. Verify the Pod’s Container Topology

The quickest way to confirm if the sidecar was injected is to check the container count on the failed or running pod.

# Get the pod name and container status
kubectl get pods -n <your-namespace> | grep <pipeline-pod-prefix>

How to interpret the output:

  • STATUS: Completed (1/2) or Running (2/2): The pod contains the main code container and the Argo wait container. The Istio sidecar is missing.

  • STATUS: Running (3/3): The pod contains main, wait, and istio-proxy. The sidecar is present; your network issue is likely DNS or a bad URL.

2. Inspect the Pod’s Explicit Annotations

Even if you added the annotation in your Python code, you should verify that MLOps Studio successfully translated it to the Kubernetes Pod spec.

# Extract the pod's annotations
kubectl get pod <pod-name> -n <your-namespace> -o jsonpath='{.metadata.annotations}' | jq .

What to look for: Search the JSON output for “sidecar.istio.io/inject”: “true”. If it is missing, the KFP compiler or Argo controller stripped it, or the kubernetes.add_pod_annotation function was not applied correctly in the pipeline definition.

3. Test Network Reachability from Inside the Pod

To prove that the mesh is blocking direct access, you can execute an interactive shell inside the main container of a running (or suspended) pipeline pod and test the connection manually.

# Exec into the main container to test curl
kubectl exec -it <pod-name> -n <namespace> -c main -- bash

# Inside the pod, try to hit the internal Model Registry service (provide the correct cluster URL)
curl -v http://model-registry-service.svc.cluster.local:8080/api/model_registry/v1alpha3/registered_models

Diagnosing the curl output

  • Result: Connection timed out or Could not resolve host

    Why: Without the istio-proxy capturing and routing the outbound DNS/TCP request, the standard Kubernetes DNS or network policy is dropping the traffic because it expects all communication to the registry to be encrypted via mTLS.

  • Result: HTTP 503 Service Unavailable or Upstream connect error

    Why: You reached the Istio ingress gateway for the registry service, but because you lack the Envoy sidecar, your request does not contain the required Istio headers or mTLS certificates. The mesh actively rejected you.