AI System Design Interview Questions: ChatGPT, RAG, LLM Inference, and Agents

System design interviews are changing.

Traditional questions such as “Design Twitter,” “Design Uber,” and “Design YouTube” are still important. They test whether you understand databases, caching, partitioning, replication, messaging, and high availability.

But engineers working on modern platforms now encounter a different category of problem:

Design a ChatGPT-like conversational assistant.
Design a retrieval-augmented generation system.
Design an LLM inference platform.
Design an AI agent that can call external tools.
Design an enterprise AI assistant for private documents.
Design an evaluation platform for generative AI applications.

These questions still require classical distributed-systems knowledge. An AI product needs APIs, queues, storage, authentication, observability, rate limiting, and reliable deployment.

The difference is that it also introduces expensive accelerators, probabilistic output, long-running requests, model routing, vector retrieval, prompt construction, safety controls, and quality evaluation.

This guide explains the most important AI system design interview questions and what a strong candidate should discuss for each.

For a broader preparation roadmap covering traditional and modern problems, see 64 System Design Interview Questions, Ranked From Easiest to Hardest.

Why AI System Design Is Different

A conventional service usually transforms an input into a deterministic output.

If a user requests order number 123, the service should retrieve order 123. Two identical requests should usually return the same underlying information.

Generative AI systems behave differently.

A model may produce different responses to the same prompt. A response can be grammatically convincing while being factually wrong. Latency depends on the number of generated tokens. Serving capacity is constrained by accelerator memory, not merely CPU utilization. Product quality may depend on prompts, retrieved context, model versions, safety filters, and external tools.

This creates several new design dimensions.

1. Quality is part of the architecture

Traditional systems are often measured using availability, latency, throughput, and error rate.

AI systems need those metrics, but they also need measures such as:

Answer correctness
Relevance
Groundedness
Retrieval quality
Hallucination rate
Tool-use success
Safety-policy compliance
User satisfaction

A system that returns a response in 200 milliseconds is not useful if that response is wrong.

2. Requests are computationally expensive

An ordinary API server may process thousands of lightweight requests per second.

An LLM request can occupy expensive GPU memory while processing a long prompt and generating hundreds of tokens. The architecture must therefore optimize batching, memory utilization, model placement, and request scheduling.

3. Latency is experienced as a stream

Users do not normally wait for an entire answer before seeing anything. Tokens are streamed as they are generated.

This introduces at least two important latency measurements:

Time to first token: How quickly generation begins.
Inter-token latency: How smoothly subsequent tokens arrive.

A system may have acceptable total latency but still feel slow if the first token takes too long.

4. Data enters the system in several ways

An AI application may depend on:

Model-training data
User prompts
Conversation history
Retrieved documents
Tool results
Feedback
Evaluation datasets
Safety policies

Each data type has distinct requirements for retention, privacy, freshness, and consistency.

5. Failure is not always binary

A traditional request may succeed or fail.

An AI request can technically succeed but produce a low-quality answer, retrieve the wrong documents, call the wrong tool, exceed a cost budget, or violate a safety rule.

The architecture must detect and respond to these softer failure modes.

A Framework for Answering Any AI System Design Question

Before considering individual questions, use a consistent interview structure.

Step 1: Clarify the product

Ask what the system is expected to do.

For example:

Is the assistant general-purpose or domain-specific?
Does it need private enterprise data?
Can it take actions or only provide answers?
Does it support text only, or also images, audio, and files?
Are responses expected in real time?
Does the system need citations?
Which decisions require human approval?

Without this clarification, “Design an AI assistant” is too broad.

Step 2: Define scale and service-level objectives

Estimate:

Daily and peak requests
Average prompt size
Average output length
Concurrent users
Required time to first token
Model size
GPU-memory requirements
Availability target
Cost per request

AI systems are often constrained by cost as much as by technical capacity.

Step 3: Separate the application layer from the model layer

The application layer may include:

Authentication
Billing
Conversation history
File management
User preferences
Rate limiting
Analytics

The AI layer may include:

Prompt construction
Retrieval
Model routing
Inference scheduling
Safety checks
Tool execution
Evaluation

Keeping these concerns separate makes the design easier to explain and evolve.

Step 4: Trace the complete request path

Describe what happens from the moment a user submits a prompt until the final result is displayed.

A typical path may be:

Authenticate the request.
Enforce quotas.
Load conversation state.
Retrieve relevant context.
Construct the model prompt.
Run input-safety checks.
Select a model.
Schedule inference.
Stream tokens.
Run output-safety checks.
Store the response.
Record metrics and feedback.

Step 5: Discuss failure, quality, and cost

A strong answer should explain:

What happens when the primary model is overloaded?
What happens when retrieval returns no useful documents?
How are duplicate tool calls prevented?
How does the system degrade gracefully?
How are model changes evaluated?
How is tenant data isolated?
How are expensive requests controlled?

These discussions distinguish a production design from a demo.

Question 1: Design ChatGPT

A ChatGPT-like system is one of the most comprehensive AI system design questions.

The functional requirements may include:

Starting a conversation
Sending prompts
Receiving streamed responses
Viewing conversation history
Regenerating an answer
Uploading files
Choosing among models
Enforcing free and paid usage limits

A useful high-level architecture contains the following components.

API gateway

The gateway handles authentication, request routing, rate limiting, quotas, and basic validation.

Long-running generation requests may use Server-Sent Events or WebSockets to stream tokens to clients.

Conversation service

This service manages:

Conversations
Messages
User preferences
Message ordering
Conversation titles
Retention and deletion

Conversation metadata can live in a transactional database, while large attachments may be placed in object storage.

Context builder

Models have finite context windows. The context builder decides what information should be included in the next request.

It may combine:

The system prompt
Recent conversation messages
A summary of older messages
Retrieved documents
User preferences
Tool outputs

Simply sending the entire conversation forever is expensive and eventually impossible. Older content may need summarization or selective retrieval.

Model gateway

The model gateway provides a single interface to multiple model backends.

It can route requests based on:

Task type
Required quality
User subscription
Context length
Latency target
Current capacity
Cost budget
Model availability

A simple request may use a smaller, faster model, while complex reasoning may be routed to a more capable one.

Inference scheduler

The scheduler assigns requests to model replicas running on accelerators.

It should consider:

Available GPU memory
Model placement
Prompt length
Output-token budget
Priority
Batch compatibility
Tenant quotas

A naive first-in, first-out scheduler can allow a few extremely long prompts to delay many short requests.

Streaming layer

Generated tokens should be forwarded incrementally to the user.

The system must also handle:

Client disconnections
User cancellation
Partial responses
Network retries
Moderation during generation
Final persistence after streaming completes

Safety layer

Input and output policies may detect:

Prompt injection
Sensitive information
Disallowed requests
Malicious files
Unsafe tool instructions
Data leakage

Safety should not be treated as one filter placed at the end. Different checks may be required before retrieval, before tool execution, before inference, and before returning the final response.

Important deep dives

An interviewer may ask:

How would you reduce time to first token?
How would you support 100 million users?
How would you prevent one tenant from consuming all GPU capacity?
How would you summarize long conversations?
How would you route between multiple models?
How would you preserve availability during GPU shortages?
How would you limit the cost for free users?

The Design ChatGPT walkthrough provides a structured example of this problem.

Question 2: Design a RAG System

Retrieval-augmented generation, or RAG, allows a model to answer using information retrieved from external sources.

A common interview prompt is:

Design an enterprise assistant that answers employee questions using internal documents and provides citations.

A RAG system has two major paths:

The ingestion path
The query path

The ingestion path

Documents may come from file uploads, internal wikis, cloud drives, databases, or support systems.

The ingestion pipeline performs several stages.

Document extraction

Files must be converted into usable text.

The system may need parsers for:

PDFs
Word documents
Presentations
HTML pages
Spreadsheets
Scanned images

The extraction process should preserve useful metadata such as titles, headings, page numbers, owners, and access permissions.

Chunking

Long documents are divided into smaller segments.

Chunks that are too large may contain irrelevant text and consume excessive context. Chunks that are too small may lose meaning.

Possible strategies include:

Fixed token windows
Paragraph-based chunking
Heading-aware chunking
Overlapping windows
Semantic chunking

There is no universally correct chunk size. It should be tested against representative questions.

Embedding generation

Each chunk is converted into a numerical vector using an embedding model.

The embedding service should be versioned because changing models can require re-embedding the entire corpus.

Indexing

The system stores:

Embeddings
Original text
Document metadata
Access-control information
Source location
Embedding version
Update timestamp

A vector index enables semantic retrieval. A traditional inverted index can support keyword retrieval. Many production systems combine both.

The query path

When a user submits a question:

Authenticate the user.
Generate an embedding for the query.
Retrieve candidate chunks.
Apply access-control filtering.
Rerank the candidates.
Select the best context.
Construct the prompt.
Generate the answer.
Attach citations.
Evaluate or log the result.

Hybrid retrieval

Semantic retrieval is useful when the query and source use different words with similar meanings.

Keyword retrieval is useful for exact terms such as:

Product codes
Error messages
Names
Dates
Identifiers

Combining both methods often produces better coverage.

Reranking

Vector similarity may retrieve documents that are generally related but not directly useful.

A reranker can score the top candidates more accurately before they are sent to the LLM. This improves answer quality while keeping the final prompt small.

Access control

Security is one of the most important parts of enterprise RAG.

A user should never retrieve a document they are not authorized to view. Filtering after the model has already received the document is too late.

Permissions should be enforced during retrieval, with tenant and user identity included in the query path.

Freshness and deletion

The system must react when:

A document changes.
A document is deleted.
Permissions change.
A user loses access.
A newer policy replaces an older one.

The ingestion pipeline may use event-driven updates, periodic crawling, or both.

RAG evaluation

A RAG system should separately evaluate:

Retrieval quality: Did the system find the relevant document?
Generation quality: Did the model use the retrieved context correctly?
Citation quality: Do the cited sources actually support the answer?

This separation is important. A poor answer can result from failed retrieval even when the model behaves correctly.

Question 3: Design an LLM Inference Platform

This question focuses less on the product interface and more on the infrastructure that serves models.

A possible prompt is:

Design a multi-tenant platform that serves several large language models to millions of requests.

The platform may need to support:

Multiple model families
Different model sizes
Streaming generation
Priority tiers
Autoscaling
Usage accounting
Model versioning
Regional deployment
Fine-tuned adapters

Inference gateway

The gateway exposes a consistent API and performs:

Authentication
Quota enforcement
Request validation
Model selection
Token-limit checks
Admission control
Cost estimation

Admission control is critical. Accepting unlimited work and allowing it to queue indefinitely creates poor latency and can destabilize the system.

Model registry

The registry tracks:

Model version
Artifact location
Supported hardware
Memory requirements
Context length
Quantization format
Deployment status
Safety and evaluation results

Rollouts should use immutable versions so requests and incidents can be traced to the exact model that served them.

Model placement

Loading a large model into GPU memory can take substantial time. The scheduler cannot treat models like lightweight stateless application containers.

It must decide:

Which models remain loaded
How many replicas each model receives
Where fine-tuned adapters are placed
When models should be unloaded
How capacity is distributed across regions

Popular models may remain warm, while rarely used models may accept a cold-start delay.

Prefill and decode

LLM inference contains two different computational phases.

Prefill processes the input prompt and can often benefit from parallel computation.

Decode generates tokens sequentially and is usually memory-bandwidth intensive.

Separating or independently scheduling these phases can improve utilization, but it also adds network and orchestration complexity.

Continuous batching

Instead of waiting for a fixed group of requests to finish together, continuous batching adds and removes requests dynamically as generation progresses.

This improves GPU utilization, especially when responses have different lengths.

The scheduler must still prevent long requests from starving shorter ones.

KV cache

The key-value cache stores intermediate attention state so the model does not recompute the entire prompt for every generated token.

KV-cache management affects:

Maximum concurrency
Memory pressure
Long-context support
Prefix reuse
Request eviction

A shared prompt prefix—such as a large system prompt—may sometimes be cached and reused across compatible requests.

Scaling

GPU utilization alone may not be sufficient for autoscaling.

Useful signals include:

Queue length
Time to first token
Tokens generated per second
KV-cache pressure
Number of active sequences
Predicted token demand
Model-specific backlog

Because accelerator provisioning may be slow, the platform may need reserved capacity and predictive scaling.

Graceful degradation

When capacity is limited, the system may:

Route to a smaller model.
Reduce the maximum output length.
Reject low-priority requests.
Disable expensive features.
Queue batch workloads.
Move traffic to another region.
Use a third-party model provider.

A strong interview answer discusses the quality and cost consequences of each fallback.

Question 4: Design an AI Agent Platform

An AI agent does more than produce text. It can plan a sequence of actions, call tools, observe results, update its state, and continue until a goal is completed.

A typical prompt might be:

Design an enterprise agent that can search internal documents, update tickets, send emails, and request human approval for sensitive actions.

Core components

Agent orchestrator

The orchestrator controls the execution loop:

Receive a goal.
Construct the current context.
Ask the model for the next action.
Validate the proposed action.
Execute the selected tool.
Store the result.
Decide whether to continue.
Produce the final response.

The orchestrator—not the model—should enforce hard limits such as maximum steps, timeouts, budgets, and approval requirements.

Tool registry

The tool registry describes each available capability:

Tool name
Purpose
Input schema
Required permissions
Timeout
Retry policy
Risk level
Whether human approval is required

Tool definitions should be versioned because changing their schemas can break existing agent behavior.

Tool execution service

Tool calls should run through controlled executors rather than allowing the model unrestricted access to internal systems.

The executor handles:

Authentication
Input validation
Secrets
Network policy
Timeouts
Retries
Audit logging
Output normalization

High-risk operations should use narrow, purpose-built APIs.

State and memory

Agents may need several kinds of memory.

Working memory contains the current task, observations, and intermediate steps.

Session memory preserves information during one user interaction.

Long-term memory stores information across sessions.

External memory may contain documents retrieved from databases or vector indexes.

Not everything should be stored forever. Memory needs explicit retention, privacy, and deletion policies.

Human approval

Actions such as sending payments, deleting data, publishing content, or modifying production systems should not be executed solely because a model requested them.

The agent can create a proposed action, pause its workflow, and wait for authorized approval.

The approval record should contain:

The intended action
The relevant parameters
Why it was proposed
The expected effect
The identity of the approver
An expiration time

Idempotency

Agents may retry actions after timeouts.

Without idempotency, a retry could send the same email twice, create duplicate tickets, or repeat a transaction.

Every state-changing tool call should include a stable execution identifier or idempotency key.

Agent-specific failure modes

A strong candidate should discuss:

Infinite planning loops
Repeated tool calls
Prompt injection inside retrieved content
Tool hallucination
Stale observations
Excessive cost
Partial workflow completion
Conflicting actions
Unauthorized data access

The system should impose:

Maximum step counts
Token budgets
Time limits
Per-tool permissions
Human checkpoints
Detailed audit logs
Recovery or compensation workflows

The Grokking Modern AI Fundamentals course can provide additional background on agentic AI, planning, memory, and tool-based behavior.

Additional AI System Design Questions to Practice

The four core questions cover much of the modern AI stack, but interviewers can frame the same concepts in narrower ways.

Design an enterprise AI copilot

Focus on tenant isolation, document permissions, RAG, conversation history, model routing, auditability, and data retention.

Design a coding assistant

Discuss repository indexing, code-aware chunking, low-latency suggestions, context selection, IDE integration, private-code protection, and evaluation of generated code.

Design an AI evaluation platform

Cover dataset versioning, offline evaluation, human review, model comparison, prompt experiments, regression detection, and production feedback.

Design a multi-model gateway

Explain routing between internal and third-party models based on cost, quality, latency, privacy, context length, and availability.

Design a semantic-search platform

Focus on ingestion, embeddings, vector indexes, hybrid retrieval, filtering, reranking, index updates, and relevance metrics.

Design an AI safety and guardrails service

Discuss policy versioning, input and output classification, prompt-injection detection, tool restrictions, personally identifiable information, appeals, and false-positive handling.

Design a prompt-management platform

Cover prompt templates, version control, experiments, rollout, rollback, tenant overrides, caching, and compatibility with changing model versions.

Design a multimodal assistant

Add image, audio, and document ingestion, media storage, preprocessing, modality-specific models, content safety, and larger payload management.

What Interviewers Look for in AI System Design Answers

A weak answer places an LLM box in the center of a diagram and connects it to an API.

A strong answer explains the system around the model.

Interviewers want to see whether you can reason about:

End-to-end architecture

Can you connect the client, application services, retrieval layer, model platform, storage, and observability systems?

Trade-offs

Can you compare:

Larger models versus smaller models
Quality versus latency
Quality versus cost
Long context versus retrieval
Hosted APIs versus self-hosting
Semantic retrieval versus keyword search
Autonomy versus human control

Reliability

Can the system continue operating when a model, vector index, tool, region, or third-party provider is unavailable?

Evaluation

Can you tell whether a new model or prompt actually improved the product?

Security

Can you protect tenant data, prevent unauthorized retrieval, constrain tool use, and manage sensitive prompts?

Cost

Can you estimate and control token usage, accelerator capacity, retrieval cost, storage, and third-party API spending?

The model is only one component. Production readiness comes from the architecture surrounding it.

How to Prepare

Candidates new to large-scale architecture should first learn the traditional foundations.

Grokking System Design Fundamentals introduces the core building blocks behind scalable systems, including databases, caches, queues, replication, partitioning, and load balancing.

The original Grokking the System Design Interview applies those concepts to common interview problems and teaches a structured way to move from requirements to architecture and trade-offs.

The System Design Interview Crash Course is useful for practicing a consistent interview framework across modern case studies, including a complete ChatGPT design problem.

Engineers preparing for senior and staff-level discussions can continue with Advanced System Design Interview, Volume II, which emphasizes open-ended problems, failures, and defensible architectural decisions.

Grokking Scalable Systems for Interviews is a useful next step for strengthening scalability, observability, fault tolerance, and performance reasoning.

For every AI design problem, practice three times:

Design the happy path.
Design for failure and overload.
Defend the quality, safety, and cost trade-offs.

That third pass is where most of the valuable interview discussion occurs.

Final Takeaway

AI system design is not a replacement for traditional system design.

It is a traditional system design combined with a new set of constraints.

You still need to understand APIs, storage, caching, queues, partitioning, replication, security, observability, and fault tolerance.

But you must now apply those concepts to systems with:

Probabilistic outputs
Expensive inference
Streaming generation
Vector retrieval
Dynamic prompts
Long-lived context
External tools
Model evaluation
Safety requirements
Human approval

Start with four foundational problems:

Design ChatGPT.
Design a RAG platform.
Design an LLM inference service.
Design an AI agent platform.

Master the request flow, deep dives, failure modes, and trade-offs behind each one.

Once you can explain those systems clearly, most other AI system design questions become variations of the same underlying building blocks.