--- name: gke-inference description: >- Deploys and optimizes AI/ML inference workloads on GKE, using GPUs, TPUs, and model servers. Use when deploying GKE inference servers, configuring GKE GPU resources for inference, or deploying LLMs on GKE. Don't use for generic batch jobs or HPC task queues (use gke-batch-hpc instead). metadata: category: Containers --- # GKE AI/ML Inference This reference covers deploying AI/ML inference workloads on GKE using Google's Inference Quickstart (GIQ) and best practices for LLM serving. > **MCP Tools:** `apply_k8s_manifest`, `get_k8s_resource`, `get_k8s_logs`, > `get_k8s_rollout_status`, `describe_k8s_resource`, `list_k8s_events`. > **CLI-only:** `gcloud container ai profiles *` ## When to Use - Deploy an AI model (Llama, Gemma, Mistral, etc.) to GKE - Generate optimized Kubernetes manifests for inference - Select GPU/TPU accelerators for model serving - Configure autoscaling for LLM inference ## Prerequisites - A golden path GKE Autopilot cluster (GPU workloads are supported via ComputeClasses and NAP) - `gcloud` CLI authenticated - Sufficient GPU/TPU quota in the target region ## Workflow ### 1. Discovery: Find Models and Hardware ```bash # List all supported models gcloud container ai profiles models list --quiet # Find valid accelerator/server combinations for a model gcloud container ai profiles list --model= --quiet # Example: what can run Gemma 2 9B? gcloud container ai profiles list --model=gemma-2-9b-it --quiet ``` ### 2. Generate Manifest ```bash gcloud container ai profiles manifests create \ --model= \ --model-server= \ --accelerator-type= \ --target-ntpot-milliseconds= --quiet > inference.yaml ``` **Parameters:** - `--model`: Model ID (e.g., `gemma-2-9b-it`, `llama-3-8b`) - `--model-server`: Inference server (`vllm`, `tgi`, `triton`, `tensorrt-llm`) - `--accelerator-type`: GPU/TPU type (`nvidia-l4`, `nvidia-tesla-a100`, `nvidia-h100-80gb`) - `--target-ntpot-milliseconds`: Target Normalized Time Per Output Token (optional, for latency optimization) **Example:** ```bash gcloud container ai profiles manifests create \ --model=gemma-2-9b-it \ --model-server=vllm \ --accelerator-type=nvidia-l4 \ --target-ntpot-milliseconds=50 --quiet > inference.yaml ``` ### 3. Review and Deploy ```bash # Review for placeholders (HF tokens, PVCs) cat inference.yaml # Deploy kubectl apply -f inference.yaml # Monitor kubectl get pods -w kubectl logs -f ``` > Some models require Hugging Face tokens. Create a Kubernetes Secret and > reference it in the manifest. ## GPU ComputeClass for Inference For Autopilot clusters, create a ComputeClass to target GPU nodes: ```yaml apiVersion: cloud.google.com/v1 kind: ComputeClass metadata: name: l4-inference spec: priorities: - machineFamily: g2 gpu: type: nvidia-l4 count: 1 minCores: 4 minMemoryGb: 16 ``` ## Accelerator Selection Guide | Accelerator | Best For | Memory | Relative Cost | | ------------------- | ------------------------ | ----------- | ------------- | | NVIDIA T4 | Budget inference, | 16 GB | Lowest | : : lightweight legacy : : : : : models : : : | NVIDIA L4 (G2) | Small-medium model | 24 GB | Low | : : inference, video, : : : : : graphics : : : | NVIDIA RTX PRO 6000 | Multimodal AI, | 96 GB | Medium | : (G4) : high-fidelity 3D, : : : : : fine-tuning : : : | Cloud TPU v5e | Cost-effective | Varies | Medium | : : transformer inference : : : | Cloud TPU v5p | High-performance | Varies | High | : : training : : : | Cloud TPU v6e | High-efficiency next-gen | 32 GB/chip | Medium-High | : (Trillium) : training & serving : : : | Cloud TPU v7x | Ultra-scale inference & | 192 GB/chip | High | : (Ironwood) : agentic workflows : : : | NVIDIA A100 | Large model inference, | 40/80 GB | High | : : enterprise ML : : : | NVIDIA H100 / H200 | Frontier model training, | 80/141 GB | Highest | : : high throughput : : : | NVIDIA B200 (A4) | Blackwell-scale | 192 GB | Highest | : : training, FP4 precision : : : | NVIDIA GB200 (A4X) | Rack-scale AI (Grace | Massive | Highest | : : Blackwell Superchip) : : : ## Autoscaling LLM Inference ### GPU-based autoscaling Use custom metrics for GPU utilization: ```yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: llm-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: llm-server minReplicas: 1 maxReplicas: 10 metrics: - type: Pods pods: metric: name: gpu_duty_cycle target: type: AverageValue averageValue: "80" ``` ### Best practices for inference autoscaling 1. **Use DCGM metrics**: Golden path enables DCGM monitoring for GPU utilization metrics 2. **Set appropriate minReplicas**: At least 1 for always-on serving; 0 for batch/on-demand 3. **Tune scale-down delay**: LLM model loading is slow; use longer stabilization windows 4. **Consider queue depth**: Scale on pending requests rather than pure GPU utilization for latency-sensitive workloads ## Optimization Tips - **Quantization**: Use quantized models (GPTQ, AWQ) to reduce GPU memory and increase throughput - **Batching**: Configure model server batch size for throughput vs latency trade-off - **Tensor parallelism**: Split large models across multiple GPUs within a node - **KV cache optimization**: Tune `--gpu-memory-utilization` in vLLM for KV cache allocation ## Troubleshooting | Issue | Cause | Fix | | ------------------ | ------------------------ | --------------------------- | | Invalid | Unsupported tuple | Re-run `gcloud container ai | : model/accelerator : : profiles list : : combination : : --model=` : | GPU quota exceeded | Regional quota limit | Request quota increase or | : : : try a different region : | OOM on GPU | Model too large for | Use larger GPU, enable | : : accelerator : quantization, or use tensor : : : : parallelism : | Slow cold start | Large model loading from | Use local SSD for model | : : registry : caching; pre-pull images :