KServe

KServe is an open-source, cloud-agnostic platform that simplifies the deployment and serving of machine learning (ML) and generative AI models on Kubernetes. It provides a standardized API and framework for running models from various ML toolkits at scale. 

How KServe works

KServe provides a Kubernetes Custom Resource Definition (CRD) called InferenceService to make deploying and managing models easier. A developer specifies their model’s requirements in a YAML configuration file, and KServe automates the rest of the process. 

The platform has two key architectural components: 

• Control Plane: Manages the lifecycle of the ML models, including versioning, deployment strategies, and automatic scaling.
• Data Plane: Executes the inference requests with high performance and low latency. It supports both predictive and generative AI models and adheres to standardized API protocols. 

Key features

• Standardized API: Provides a consistent interface for different model types and frameworks, promoting interoperability.
• Multi-framework support: KServe supports a wide range of ML frameworks, including:
  • TensorFlow
  • PyTorch
  • Scikit-learn
  • XGBoost
  • Hugging Face (for large language models)
  • NVIDIA Triton (for high-performance serving)
• Flexible deployment options: It supports different operational modes to fit specific needs:
  • Serverless: Leverages Knative for request-based autoscaling and can scale down to zero when idle to reduce costs.
  • Raw Deployment: A more lightweight option without Knative, relying on standard Kubernetes for scaling.
  • ModelMesh: An advanced option for high-density, multi-model serving scenarios.
• Advanced deployment strategies: KServe enables sophisticated rollouts for production models, including:
  • Canary rollouts: Gradually shifting traffic from an old model version to a new one.
  • A/B testing: Routing traffic between different model versions to compare their performance.
  • Inference graphs: Building complex pipelines that can combine multiple models or perform pre/post-processing steps.
• Scalability and cost efficiency: By automatically scaling model instances up or down based on traffic, KServe optimizes resource usage and costs, especially with its scale-to-zero capability. 

Core components

KServe is often used in combination with other cloud-native technologies to provide a complete solution: 

• Kubernetes: The foundation on which KServe operates, managing containerized model instances.
• Knative: An optional but commonly used component that provides the serverless functionality for request-based autoscaling.
• Istio: A service mesh that provides advanced networking, security, and traffic management capabilities, such as canary deployments.
• ModelMesh: An intelligent component used for high-density, multi-model serving by managing the loading and unloading of models from memory.