A single A10 GPU on OCI costs $1.52/hr. Running 24/7, that's $1,094/month. For a production inference service with steady traffic, that's fine. But I had a staging environment and a couple of internal tools that got maybe 20 requests per day. I was paying over $2,000/month for GPUs that sat idle 95% of the time.

The obvious solution: scale to zero when there's no traffic, spin up when a request comes in. KEDA does this on Kubernetes, but getting it to work properly with GPU pods took some figuring out.

Why Scaling GPUs Is Harder Than Scaling CPU Pods

With normal HTTP services, KEDA watches a metric (HTTP requests, queue depth, whatever), and Kubernetes can spin up a new pod in seconds. The user barely notices.

GPU pods are different:

1. Cold start is slow - Model loading takes 60-120 seconds
2. Node scaling is slow - If there's no GPU node in the pool, OKE needs to provision a new VM (3-5 minutes)
3. Images are huge - 5-15GB pull times if not cached
4. GPU resources are binary - You can't give a pod "half a GPU" (without MIG)

So you can't just scale-to-zero and expect sub-second response times when traffic returns. The trade-off is cost savings vs. cold start latency. For my use case (internal tools, staging), a 2-3 minute cold start was acceptable.

The Setup

1. Install KEDA on OKE

helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda \
  --namespace keda-system \
  --create-namespace

2. Prometheus for Request Metrics

I'm using the nginx ingress controller's Prometheus metrics to track request rate. If you're using OCI's native load balancer, you'd use OCI Monitoring metrics instead.

# prometheus-scaledobject.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-scaler
  namespace: inference
spec:
  scaleTargetRef:
    name: vllm-inference
  minReplicaCount: 0          # scale to zero
  maxReplicaCount: 3
  cooldownPeriod: 300          # wait 5 min of no traffic before scaling down
  pollingInterval: 15

  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        metricName: http_requests_total
        query: |
          sum(rate(nginx_ingress_controller_requests{
            namespace="inference",
            service="vllm-inference"
          }[2m]))
        threshold: "1"         # scale up if >1 req/sec averaged over 2 min
        activationThreshold: "0.1"  # activate from zero if any traffic

The key settings:

• minReplicaCount: 0 — this is what enables scale-to-zero
• cooldownPeriod: 300 — 5 minutes of no traffic before scaling down (prevents flapping)
• activationThreshold: "0.1" — even a trickle of traffic triggers scale-up from zero

3. Handling the Cold Start

When the pod scales from zero, there's a gap. The request that triggered the scale-up needs to wait for the pod to be ready. I handle this with a simple queue pattern:

# queue-proxy.yaml — lightweight proxy that holds requests during cold start
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-proxy
  namespace: inference
spec:
  replicas: 1    # always running, tiny resource footprint
  template:
    spec:
      containers:
        - name: proxy
          image: iad.ocir.io/mytenancy/inference-proxy:v1
          ports:
            - containerPort: 8080
          env:
            - name: BACKEND_URL
              value: "http://vllm-inference:8000"
            - name: TIMEOUT_SECONDS
              value: "180"    # wait up to 3 min for backend
          resources:
            requests:
              cpu: 50m
              memory: 64Mi

The proxy is a tiny Go service (always running, costs almost nothing) that:

• Accepts incoming requests immediately
• Forwards them to the vLLM backend
• If the backend isn't ready, retries with exponential backoff up to 3 minutes
• Returns a 503 with "model loading, please retry" if it times out

func proxyHandler(w http.ResponseWriter, r *http.Request) {
    backendURL := os.Getenv("BACKEND_URL")
    timeout, _ := strconv.Atoi(os.Getenv("TIMEOUT_SECONDS"))

    deadline := time.Now().Add(time.Duration(timeout) * time.Second)
    backoff := 2 * time.Second

    for time.Now().Before(deadline) {
        resp, err := http.DefaultClient.Do(cloneRequest(r, backendURL))
        if err == nil {
            copyResponse(w, resp)
            return
        }
        time.Sleep(backoff)
        backoff = min(backoff*2, 15*time.Second)
    }

    http.Error(w, "inference backend unavailable, try again shortly", 503)
}

4. Keeping a Warm GPU Node

The slowest part of cold start isn't model loading — it's waiting for OKE to provision a GPU node when none exist. This takes 3-5 minutes.

My workaround: keep one GPU node always available, but let the inference pods on it scale to zero. The node costs money even when idle, but it's a single node vs. multiple. And when traffic comes in, the pod starts in ~90 seconds (model loading) instead of 5+ minutes (node provisioning + model loading).

# GPU node pool with min 1 node (always warm)
oci ce node-pool update \
  --node-pool-id $GPU_NODE_POOL_ID \
  --node-config-details '{
    "size": 1,
    "placementConfigs": [...]
  }'

For staging environments where the 5-minute cold start is acceptable, I set the node pool to autoscale from 0 to 2 nodes and let OKE handle it.

The Savings

My three GPU workloads (staging vLLM, internal summarizer, internal code review tool) were running 24/7 on three A10 instances:

65% savings. The internal tools scale up when someone uses them (a few times a day) and scale back down after 5 minutes of idle. The warm node means cold starts are 90 seconds, which is fine for internal users.

When Not to Do This

• Customer-facing APIs - 90-second cold starts are unacceptable. Keep replicas warm.
• Steady traffic - If your GPU is busy >60% of the time, scale-to-zero doesn't save enough to justify the complexity.
• Real-time applications - Anything latency-sensitive should stay always-on.

This works for internal tools, batch endpoints, staging environments, and anything where "please wait a moment" is an okay response.

Pavan Madduri — Oracle ACE Associate, CNCF Golden Kubestronaut. I'm also building keda-gpu-scaler for GPU-aware autoscaling. GitHub | LinkedIn | Website | Google Scholar | ResearchGate