How to Test LLM-Powered Applications Effectively
Testing a CRUD app is deterministic. You input X, you expect Y, you assert equality. Testing an LLM-powered application is different in a way that breaks most of your existing instincts.
The model's output is probabilistic. The same prompt can return different phrasing across runs. "Correct" is often subjective. Traditional assertEqual doesn't work here.
Here's how to think about testing LLM apps properly.
The Three Layers of an LLM App
Before writing a single test, map out what you're actually testing:
[ User Input ]
↓
[ Prompt Construction ] ← Layer 1: Deterministic. Testable normally.
↓
[ LLM API Call ] ← Layer 2: Non-deterministic. Mock in unit tests.
↓
[ Output Parsing ] ← Layer 3: Deterministic. Testable normally.
↓
[ App Response ]
Most bugs aren't in the LLM — they're in layers 1 and 3. Start there.
Layer 1: Test Prompt Construction
Your prompt builder is plain code. Test it like code.
def build_prompt(user_query: str, context: str) -> str:
return f"""You are a helpful assistant.
Context: {context}
User: {user_query}
Answer concisely."""
def test_prompt_includes_context():
prompt = build_prompt("What is the policy?", "Refund window is 30 days.")
assert "Refund window is 30 days." in prompt
def test_prompt_has_system_instruction():
prompt = build_prompt("Hi", "")
assert "You are a helpful assistant" in prompt
These are fast, free, and catch the majority of regressions.
Layer 2: Test Output Parsing
If your app parses structured data from LLM output, test the parser independently with canned responses:
def parse_llm_json_response(raw: str) -> dict:
import json, re
match = re.search(r'\{.*\}', raw, re.DOTALL)
if not match:
raise ValueError("No JSON found in response")
return json.loads(match.group())
def test_parser_extracts_json():
raw = "Here is the result: {\"score\": 8, \"reason\": \"Clear\"}"
result = parse_llm_json_response(raw)
assert result["score"] == 8
def test_parser_raises_on_no_json():
with pytest.raises(ValueError):
parse_llm_json_response("Sorry, I cannot help with that.")
Layer 3: Evaluating LLM Output Quality
For actual model output, shift from assertion-based testing to evaluation-based testing. Three practical approaches:
1. Rubric Scoring (LLM-as-Judge)
def evaluate_response(question, answer, criteria):
eval_prompt = f"""
Rate the following answer on a scale of 1-5 for each criterion.
Question: {question}
Answer: {answer}
Criteria: {criteria}
Return JSON: {{"score": int, "reason": str}}
"""
# Call your LLM here and parse response
...
2. Semantic Similarity (for factual tasks)
from sentence_transformers import SentenceTransformer, util
model = SentenceTransformer('all-MiniLM-L6-v2')
def is_semantically_similar(expected, actual, threshold=0.85):
emb1 = model.encode(expected, convert_to_tensor=True)
emb2 = model.encode(actual, convert_to_tensor=True)
score = util.cos_sim(emb1, emb2).item()
return score >= threshold
3. Behavioral Testing (what should never happen)
FORBIDDEN_PHRASES = ["I cannot", "As an AI", "I don't have access"]
def test_no_refusals_on_valid_queries(llm_client):
response = llm_client.ask("What is the return policy?")
for phrase in FORBIDDEN_PHRASES:
assert phrase not in response, f"Got refusal: {phrase}"
Testing for Regressions: Golden Datasets
Build a golden dataset — a curated set of input/expected-output pairs — and run evaluations on every model or prompt change:
| Input | Min Score | Pass? |
|---|---|---|
| "Summarize this in 3 points" | 4/5 | ✅ |
| "Translate to French" | 4/5 | ✅ |
| "What's 2+2?" (sanity check) | 5/5 | ✅ |
This won't catch everything, but it will catch regressions — which is the main goal.
Tools Worth Knowing
- Promptfoo — open-source LLM eval framework, define test cases in YAML
- LangSmith — tracing + eval if you're on LangChain
- DeepEval — pytest-style assertions for LLM metrics
Written by Yash| Senior SDET catching failures other layers miss — cross-validating UI, API, DB simultaneously and test infrastructure.