After returning from AWS Summit London 2026 I was doing some research on running AI/ML workload in AWS EKS with Karpenter. With some assistance from Gemini I turned some of my notes from various talks into this guide that will talk through the intricacies of deploying and scaling Generative AI (GenAI) workloads on AWS EKS, leveraging the power of Karpenter.

Why GenAI/ML Infrastructure Sizing is Hard 📏

The initial challenge with GenAI/ML workloads often stems from translating business requirements into technical specifications. A request like "I have 10,000 users, and my LLM needs to respond fast" is a common starting point, but it provides insufficient detail for hardware selection. We need to convert this into a concrete workload model.

From Business Vision to Technical Metrics

LLM inference systems are rarely "sized" by a formula; they are measured and optimized. We follow a structured lifecycle to reach the optimal architecture.

---

From Estimation to Deployment: A 4-Step Framework 🗺️

To consistently meet service level objectives with the lowest possible cost, we utilise a methodical, four-step process:

---

Building the Foundation: Cluster and OS Selection 🏗️

For maximum flexibility and control over your data plane, we recommend self-managed Kubernetes clusters on Amazon EKS. AWS handles the control plan, while you manage the data plane—including networking, security, monitoring, scaling, and storage.

When choosing a Node OS, the goal is to have the smallest, most secure footprint with pre-baked drivers for your AI/ML workloads.

EKS Node Operating System Options

---

Modern Autoscaling with Karpenter ⚡

In Kubernetes, we distinguish between two types of scaling: application scaling and data plane scaling.

Kubernetes Scaling Mechanisms

Karpenter is particularly effective for AI/ML workloads due to its ability to provision the exact GPU instance needed for a pending pod in seconds, making it more dynamic and efficient than traditional Cluster Autoscaler.

When an LLM processes a request, it typically involves two phases:

1. Prefill Phase: The model processes the input tokens. No output is generated yet. Performance is measured by tokens processed per second.
2. Decode Phase: The model generates output tokens sequentially. The "Time To First Token" (TTFT) is a critical latency metric here. Large inputs and outputs can significantly increase TTFT and overall latency.

Capacity Management with Karpenter NodePools

Karpenter allows for sophisticated capacity management using NodePools, enabling precise control over your compute resources:

---

Evaluating Optimal Architecture 📊

To truly find the "sweet spot" for your GenAI architecture, thorough evaluation is essential.

Reproducible Load Testing & Experimentation

1. Design Load Tests: Use tools like Locust to create realistic load tests with real data, simulating user behavior and workload patterns.
2. Test Different Solutions: Run reproducible tests against various deployment options:
   • AWS Bedrock: Managed service for foundation models.
   • Amazon SageMaker: Fully managed service for ML lifecycle.
   • Self-managed EKS: Granular control over infrastructure.
3. Evaluate Hardware: Test against different GPU architectures (e.g., G-series, P-series) and AWS ML chips (Inferentia, Trainium).
4. Tune Runtime Parameters: Experiment with parameters like KV cache optimization and batch size to maximize performance.

AWS GPU, ML Accelerators, and FPGA Landscape 🏎️

AWS offers a diverse range of hardware tailored for different stages of the ML lifecycle:

The Trade-off: Latency vs. Throughput ⚖️

GPU sizing doesn't follow a simple formula; it's a critical trade-off between latency (how quickly a single request is processed) and throughput (how many requests are processed over time). At low concurrency, each request is fast but hardware is underutilized. As concurrency increases, hardware utilization improves, but individual request latency can suffer.

Your objective for performance testing is to find the right concurrency balance with the optimal number of compute instances and GPUs that deliver acceptable end-to-end latency and requests per second (RPS).

Case Study: Latency Requirements Directly Drive Cost

As this case study illustrates, strict latency requirements (e.g., < 2.5s) can necessitate significantly more instances and GPUs, directly impacting costs. By carefully balancing concurrency and hardware, you can find the optimal architecture that aligns with both your performance and budget goals.

---

By embracing this structured approach, you'll not only run your AI/ML workloads on AWS EKS with Karpenter, but you'll also optimize them for peak performance and cost-efficiency.