Single-model systems are simple. Multi-model systems are powerful. The challenge isn't choosing models — it's designing the architecture that orchestrates them.
A multi-model system isn't about having more models. It's about having the right model for the right task at the right time.
Architecture patterns
Five patterns cover most use cases:
| Pattern | Complexity | When to use | Tradeoff |
|---|---|---|---|
| Single Model | Lowest | Prototyping, simple tasks | Limited capability |
| Sequential | Low | Multi-step workflows | Higher latency |
| Parallel | Medium | Independent tasks | Higher cost |
| Hierarchical | High | Complex reasoning | Complex orchestration |
| Ensemble | Highest | Critical decisions | Highest cost |
Pick the simplest one that works. Complexity is real, and it compounds.
Sequential architecture
Process tasks through a chain of models, each specializing in a step.
Pattern 1: Pipeline
Pipeline pattern — each model's output feeds the next:
class ModelPipeline:
def __init__(self):
self.models = [
{"model": "qwen2.5-1.5b", "task": "classify"},
{"model": "qwen2.5-7b", "task": "extract"},
{"model": "qwen2.5-32b", "task": "reason"},
]
def process(self, input: str) -> str:
current = input
for model_config in self.models:
current = self.call_model(
model_config["model"],
self.create_prompt(model_config["task"], current)
)
return current
Latency adds up. Three models in sequence means three times the latency. Only use this when each step actually needs a different model.
Pattern 2: Router
Router pattern — classify the task, route to the specialist:
class ModelRouter:
def __init__(self):
self.classifier = "qwen2.5-1.5b"
self.specialists = {
"code": "qwen2.5-coder-7b",
"math": "qwen2.5-32b",
"creative": "claude-sonnet-4",
"general": "qwen2.5-7b",
}
def route(self, prompt: str) -> str:
task_type = self.classify(prompt)
model = self.specialists.get(task_type, self.specialists["general"])
return self.call_model(model, prompt)
The classifier is the weak link. If it misclassifies, you route to the wrong model and lose quality. Use a classifier that's good enough — even a small one works if the categories are clear.
Parallel architecture
Process independent tasks simultaneously.
Pattern 1: Fan-Out
Fan-out — run the same prompt through multiple models:
import asyncio
class ModelFanOut:
def __init__(self):
self.models = [
"qwen2.5-7b",
"qwen2.5-32b",
"claude-sonnet-4",
]
async def process(self, prompt: str) -> list[str]:
tasks = [self.call_model(model, prompt) for model in self.models]
return await asyncio.gather(*tasks)
Useful for comparison, A/B testing, or when you want to pick the best output. Expensive, but the quality gain is worth it for critical decisions.
Pattern 2: Voting
Voting — combine outputs through consensus:
class ModelVoting:
def __init__(self):
self.models = [
"qwen2.5-7b",
"qwen2.5-32b",
"claude-sonnet-4",
]
def vote(self, prompt: str) -> str:
responses = [self.call_model(model, prompt) for model in self.models]
from collections import Counter
votes = Counter(responses)
return votes.most_common(1)[0][0]
Majority voting works for classification. For generation tasks, it's harder — you need semantic similarity, not exact matches.
Hierarchical architecture
Use models at different levels of abstraction.
Pattern 1: Planner-Executor
Planner-executor — a strong model plans, smaller models execute:
class PlannerExecutor:
def __init__(self):
self.planner = "qwen2.5-32b"
self.executors = {
"code": "qwen2.5-coder-7b",
"search": "qwen2.5-7b",
"math": "qwen2.5-7b",
}
def process(self, task: str) -> str:
plan = self.call_model(self.planner, f"Plan: {task}")
results = []
for step in self.parse_plan(plan):
executor = self.executors.get(step["type"], "qwen2.5-7b")
result = self.call_model(executor, step["prompt"])
results.append(result)
return self.call_model(self.planner, f"Synthesize: {results}")
The planner does the heavy lifting. The executors handle specific tasks. This pattern works well when the planning step is expensive but the execution steps are cheap.
Pattern 2: Supervisor-Worker
Supervisor-worker — a supervisor delegates and reviews:
class SupervisorWorker:
def __init__(self):
self.supervisor = "qwen2.5-32b"
self.workers = ["qwen2.5-7b", "qwen2.5-coder-7b"]
def process(self, task: str) -> str:
assignments = self.call_model(self.supervisor, f"Assign: {task}")
results = []
for assignment in self.parse_assignments(assignments):
result = self.call_model(
assignment["worker"], assignment["task"]
)
results.append(result)
return self.call_model(self.supervisor, f"Review: {results}")
The supervisor is the bottleneck. It plans, delegates, and reviews. Make sure it's fast enough, or the whole system slows down.
Ensemble architecture
Combine multiple models for critical decisions.
Pattern 1: Weighted Ensemble
Weighted ensemble — score each model's output, pick the highest:
class WeightedEnsemble:
def __init__(self):
self.models = {
"qwen2.5-32b": 0.5,
"claude-sonnet-4": 0.3,
"qwen2.5-7b": 0.2,
}
def decide(self, prompt: str) -> str:
responses = {
model: self.call_model(model, prompt)
for model in self.models
}
scores = {}
for model, response in responses.items():
score = self.evaluate(response) * self.models[model]
scores[response] = scores.get(response, 0) + score
return max(scores, key=scores.get)
Weights reflect your confidence in each model. Adjust them based on actual performance, not benchmarks.
Pattern 2: Consensus Ensemble
Consensus ensemble — require agreement, escalate if there isn't any:
class ConsensusEnsemble:
def __init__(self, threshold: float = 0.7):
self.threshold = threshold
self.models = [
"qwen2.5-32b",
"claude-sonnet-4",
"qwen2.5-7b",
]
def decide(self, prompt: str) -> str:
responses = [
self.call_model(model, prompt)
for model in self.models
]
from collections import Counter
votes = Counter(responses)
max_votes = max(votes.values())
if max_votes / len(self.models) >= self.threshold:
return votes.most_common(1)[0][0]
return self.call_model("qwen2.5-32b", prompt)
The threshold controls how strict consensus is. 0.7 means two-thirds agreement. Lower it for faster decisions, raise it for higher confidence.
When multi-model systems make sense
Multi-model systems make sense when you have mixed workloads, need high quality for critical decisions, or are optimizing for cost or latency.
They don't make sense when all tasks are similar complexity, you're prototyping, or simplicity matters more than optimization.
The rule of thumb: start with one model. Add more when you hit a real constraint — cost, latency, or quality. Don't architect complexity before you need it.
Tradeoffs
| Pattern | Cost | Latency | Quality | Complexity |
|---|---|---|---|---|
| Single Model | Lowest | Lowest | Variable | Lowest |
| Sequential | Medium | High | High | Medium |
| Parallel | High | Low | High | Medium |
| Hierarchical | High | High | Highest | High |
| Ensemble | Highest | Medium | Highest | Highest |
Every pattern trades something. Pick the one that matches your constraints.
Related
- Model Routing Strategies — capability-based, cost-aware, latency-aware routing
- Cost Optimization for LLM Systems — token budgeting, fallback models, caching
- LLM Guardrails in Practice — input validation, output filtering, safety
- LLM Architecture — system design pillar: routing, cost, guardrails, and orchestration