Running GPU containers on Kubernetes is one of those things that works perfectly in tutorials and then breaks in confusing ways on real clusters. I've been deploying GPU workloads on OKE for a few months now, and I've built up a decent collection of debugging war stories.
This isn't a getting-started guide. This is the post I wish existed the first time I saw CrashLoopBackOff on a GPU pod with zero useful logs.
Bug 1: Pod Stuck in Pending — "0/3 nodes are available"
This was my first GPU deployment on OKE. Created a pod requesting nvidia.com/gpu: 1, and it just sat in Pending forever.
$ kubectl describe pod vllm-inference-0
Events:
Warning FailedScheduling 0/3 nodes are available: 3 Insufficient nvidia.com/gpu
Three nodes, but none had GPUs. Turns out I created the GPU node pool but it hadn't finished scaling up yet. OKE provisions GPU nodes on-demand when you create the node pool, and it takes 3-5 minutes for the instances to come up.
Fix: Just wait. But also — check that your node pool is actually using a GPU shape:
# Verify GPU nodes exist and are ready
kubectl get nodes -l nvidia.com/gpu=present
kubectl describe node <gpu-node> | grep nvidia.com/gpu
# Should show:
# nvidia.com/gpu: 1
# under Allocatable
If nvidia.com/gpu doesn't appear in Allocatable, the NVIDIA device plugin isn't running on that node. On OKE it should be automatic, but I've seen it lag behind node creation by a minute or two.
Bug 2: CUDA Version Mismatch
This one was nasty. The container started, then immediately crashed:
CUDA error: no kernel image is available for execution on the device
My Dockerfile used nvidia/cuda:12.4-runtime-ubuntu22.04, but the GPU node's driver only supported CUDA 12.2. The container runtime is newer than the host driver can handle.
Fix: Check the driver version on the node and match your CUDA image:
# SSH to GPU node or run a debug pod
kubectl run gpu-debug --image=nvidia/cuda:12.2.0-base-ubuntu22.04 \
--restart=Never --rm -it \
--overrides='{"spec":{"containers":[{"name":"gpu-debug","image":"nvidia/cuda:12.2.0-base-ubuntu22.04","command":["nvidia-smi"],"resources":{"limits":{"nvidia.com/gpu":"1"}}}]}}' \
-- nvidia-smi
The output shows the driver version and maximum CUDA version it supports. Use a CUDA runtime image that doesn't exceed that version.
On OKE, Oracle controls the GPU node image. When they update the driver, you can bump your CUDA version. Don't go the other way around.
Bug 3: OOM Killed During Model Loading
vLLM loaded halfway and then the pod got OOM killed:
$ kubectl describe pod vllm-0
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
I had resources.limits.memory: 16Gi but the model needed more during the loading phase. vLLM memory-maps the model weights, and the kernel counts that against the container's memory limit even though it's not all resident.
Fix: Set memory limits higher than you think you need, or use --gpu-memory-utilization 0.85 in vLLM to cap GPU memory usage and reduce the spill to CPU memory:
resources:
limits:
nvidia.com/gpu: 1
memory: 32Gi # was 16Gi — doubled it
requests:
memory: 16Gi
The gap between requests and limits gives the pod burst room during model loading without permanently reserving 32GB on the node.
Bug 4: Liveness Probe Restart Loop
Already mentioned this in a previous post but it's worth repeating because it gets everyone.
Model loading takes 60-120 seconds. Default liveness probe fires at 10 seconds. Kubernetes thinks the pod is dead, kills it, it restarts, starts loading again, gets killed again. Infinite loop.
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 180 # Give model time to load
periodSeconds: 15
failureThreshold: 3
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 60
periodSeconds: 10
Use a startup probe if you want to be cleaner about it:
startupProbe:
httpGet:
path: /health
port: 8000
failureThreshold: 30
periodSeconds: 10
# Gives up to 300 seconds for model loading
Bug 5: Image Pull Timeout on Large GPU Images
GPU images are 5-15GB. OCIR pull worked fine for small images but GPU images would timeout:
Failed to pull image: context deadline exceeded
Fix: Increase the kubelet image pull deadline and use imagePullPolicy: IfNotPresent so the massive image only downloads once:
spec:
containers:
- name: inference
image: iad.ocir.io/mytenancy/vllm:v1
imagePullPolicy: IfNotPresent
Also — pull from the same OCI region as your cluster. Cross-region OCIR pulls over the internet are slow. Same-region pulls go over the internal network and are 5-10x faster.
Bug 6: GPU Not Released After Pod Deletion
Deleted a pod, created a new one, and it couldn't get the GPU:
0/1 nodes are available: 1 Insufficient nvidia.com/gpu
But the old pod was gone. kubectl get pods showed nothing using the GPU.
Turns out the pod was stuck in Terminating state because the vLLM process wasn't handling SIGTERM properly. The GPU was still allocated to the zombie pod.
Fix: Add a preStop hook and set a reasonable terminationGracePeriodSeconds:
spec:
terminationGracePeriodSeconds: 30
containers:
- name: inference
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "kill -SIGTERM 1 && sleep 5"]
If a pod is truly stuck, kubectl delete pod <name> --grace-period=0 --force will release the GPU. Use it as a last resort.
My Debugging Checklist
When a GPU pod is broken on OKE, I now run through this in order:
# 1. Is the pod even scheduled?
kubectl get pod <name> -o wide
# 2. What's the event history?
kubectl describe pod <name>
# 3. Are GPU nodes ready with GPUs allocatable?
kubectl get nodes -l nvidia.com/gpu=present
kubectl describe node <gpu-node> | grep -A5 "Allocatable"
# 4. Can nvidia-smi run on the node?
kubectl exec -it <gpu-pod> -- nvidia-smi
# 5. What's the container's actual error?
kubectl logs <pod> --previous # logs from the crashed container
# 6. Resource pressure on the node?
kubectl top node <gpu-node>
90% of the time, the answer is in steps 2 or 5. The other 10% is the CUDA version mismatch, which requires step 4.
Prevention
Most of these bugs hit me once and then I added checks so they wouldn't happen again:
-
CI validates CUDA version — build step runs
nvidia-smiin a test container to verify compatibility - Startup probes on every GPU pod — no more liveness probe restart loops
- Same-region OCIR — eliminated image pull timeouts
- Memory limits 2x the model size — OOM kills are gone
None of this is complex. It's just stuff you learn by deploying real GPU workloads and watching them break.
Pavan Madduri — Oracle ACE Associate, CNCF Golden Kubestronaut. GitHub | LinkedIn | Website | Google Scholar | ResearchGate