CNCF On-Demand: Cloud Native Inference at Scale - Unlocking LLM Deployments with KServe

CNCF [Cloud Native Computing Foundation] · 2025-12-04

CNCF On-Demand: Cloud Native Inference at Scale - Unlocking LLM Deployments with KServeCNCF [Cloud Native Computing Foundation] ▶ Play the video

This item is no longer on the current shelf. Its page remains available, and editorial corrections may be made. See what is featured now.

Why it earned a slot

Finally, someone is addressing the fact that standard Kubernetes schedulers choke on token-based generation. The breakdown of the Gateway Inference Extension for token-aware routing is the technical highlight here, moving beyond simple HTTP load balancing. If you are trying to run LLMs on K8s without these specific scheduling adaptations, you are likely burning budget on idle GPUs.

The short version

Kubernetes wasn't built for text generation, and it shows.

Why it matters

As we move from PoCs to massive production deployments, the 'set it and forget it' autoscaling of standard cloud stacks fails miserably with the bursty, long-context nature of LLM traffic.

My take

Debugging GPU utilization makes clear that inference is an entirely different beast than standard serving. The deterministic scheduling mentioned here isn't just a nice-to-have; it's the only way to maintain latency SLAs without over-provisioning expensive hardware.

How it connects

• Contrast this with Ray Serve's approach to see the trade-offs between K8s-native and specialized frameworks.
• Connects directly to the vLLM and TGI backend integrations that actually make these schedulers useful.

Bottom line

Implement token-aware request routing before you scale your cluster.