aevyra-verdict
Benchmark any LLM against your data. Pick the best model, then make it better.
verdict runs your prompts across any combination of models, scores the responses with pluggable metrics, and gives you a side-by-side comparison — so you can choose the right model for your task, then track whether your prompt engineering or fine-tuning is actually moving the needle.
Use cases
Choosing the right model. Instead of guessing, run your actual prompts across GPT-5.4-mini, Claude Sonnet, Gemini, Llama — and pick the one that scores highest on your specific task.
Measuring improvement. Establish a baseline score, tweak your system prompt or fine-tune your model, re-run verdict. If the number goes up, your change helped. If it doesn't, you know to try something else.
Benchmarking open-source vs closed models. Measure how a local model stacks up against SOTA closed models on your workload — and identify exactly where the gap is.
Install
pip install aevyra-verdict
Provider SDKs are optional extras — install only what you need:
pip install aevyra-verdict[openai] # OpenAI + OpenRouter + local (Ollama/vLLM) pip install aevyra-verdict[anthropic] # Anthropic pip install aevyra-verdict[google] # Google Gemini pip install aevyra-verdict[mistral] # Mistral pip install aevyra-verdict[cohere] # Cohere pip install aevyra-verdict[all] # everything
You only need API keys for the providers you actually use.
Quick start
# 1. Check which API keys are configured aevyra-verdict providers # 2. Compare models on a dataset and save results aevyra-verdict run examples/sample_data.jsonl \ -m openai/gpt-5.4-nano \ -m qwen/qwen3.5-9b \ -o results.json # 3. Compare two local Ollama models (no API key needed) aevyra-verdict run examples/sample_data.jsonl \ -m local/llama3.1:8b \ -m local/mistral \ --base-url http://localhost:11434/v1 \ -o results.json
Or use the Python API directly:
from aevyra_verdict import Dataset, EvalRunner, RougeScore, LLMJudge from aevyra_verdict.providers import get_provider dataset = Dataset.from_jsonl("examples/sample_data.jsonl") runner = EvalRunner() runner.add_provider("openai", "gpt-5.4-nano") runner.add_provider("openrouter", "qwen/qwen3.5-9b") runner.add_metric(RougeScore()) runner.add_metric(LLMJudge(judge_provider=get_provider("openai", "gpt-5.4"))) results = runner.run(dataset) print(results.compare())
Set your API keys as environment variables (OPENAI_API_KEY, ANTHROPIC_API_KEY,
GOOGLE_API_KEY, MISTRAL_API_KEY, COHERE_API_KEY) or pass them directly when
adding providers.
How it works
The framework has four layers that compose together:
Dataset reads JSONL files where each line has a messages array (OpenAI chat
format), an optional ideal reference answer, and optional metadata for filtering.
Providers wrap each LLM API behind a common interface. The OpenAI message format
is the canonical input — each provider translates it to whatever the underlying SDK
expects (Anthropic's separate system parameter, Gemini's contents format, etc.) and
normalizes the response back into a CompletionResult with text, usage stats, and
latency.
Metrics score each response. Three families are supported:
- Reference-based (exact match, BLEU, ROUGE) — compare output against a known-good answer
- LLM-as-judge — use a separate model to evaluate quality on configurable criteria
- Custom — pass any Python function that returns a score
Runner ties it together: models and samples are dispatched concurrently via
thread pools. Rate-limit errors (HTTP 429) trigger exponential backoff with jitter
before retrying; fatal errors (auth failures, bad requests) are surfaced immediately
without burning retry budget. Results land in EvalResults.
flowchart LR
DS[Dataset]:::data
R[EvalRunner]:::model
M[Metrics]:::metric
OUT[Results]:::output
DS --> R --> M --> OUT
classDef data fill:#6E3FF3,color:#fff,stroke:none
classDef model fill:#9B6BFF,color:#fff,stroke:none
classDef metric fill:#3FBFFF,color:#fff,stroke:none
classDef output fill:#2ECC71,color:#fff,stroke:none
Usage
Dataset format
Four formats are supported. JSONL and CSV files are both accepted.
CSV — simplest format for tabular data. Column names default to input and ideal:
dataset = Dataset.from_csv("data.csv") # input + ideal columns dataset = Dataset.from_csv("data.csv", input_field="article", output_field="summary") # custom columns dataset = Dataset.from_csv("data.csv", output_field=None) # label-free
For JSONL, the format is auto-detected from the first record.
OpenAI (native):
{
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"}
],
"ideal": "The capital of France is Paris.",
"metadata": {"category": "factual", "difficulty": "easy"}
}ShareGPT (common HuggingFace fine-tuning format):
{
"conversations": [
{"from": "human", "value": "What is the capital of France?"},
{"from": "gpt", "value": "The capital of France is Paris."}
]
}Alpaca (instruction-following datasets):
{
"instruction": "Translate to French.",
"input": "Hello, how are you?",
"output": "Bonjour, comment allez-vous?"
}messages / conversations / instruction is required. ideal and metadata are
optional (or extracted automatically for ShareGPT and Alpaca). Pass format= explicitly
to override auto-detection:
dataset = Dataset.from_jsonl("sharegpt_data.jsonl", format="sharegpt") dataset = Dataset.from_jsonl("alpaca_data.jsonl", format="alpaca")
You can also create datasets inline:
dataset = Dataset.from_list([ {"messages": [{"role": "user", "content": "Hello"}], "ideal": "Hi there"}, ])
Filter by metadata fields:
hard_questions = dataset.filter(difficulty="hard", category="reasoning")
Providers
Five providers are built in:
from aevyra_verdict.providers import get_provider, list_providers print(list_providers()) # ['anthropic', 'cohere', 'google', 'mistral', 'openai'] # Each provider takes a model name and optional api_key / base_url provider = get_provider("openai", "gpt-5.4-nano", api_key="sk-...") result = provider.complete([{"role": "user", "content": "Hello"}]) print(result.text, result.latency_ms, result.usage)
The OpenAI provider works with any OpenAI-compatible API (Azure, Together, vLLM,
etc.) by passing a base_url.
To add a custom provider, subclass Provider and register it:
from aevyra_verdict.providers import Provider, register_provider class MyProvider(Provider): name = "my_provider" def complete(self, messages, temperature=0.0, max_tokens=1024, **kwargs): # your implementation ... register_provider("my_provider", MyProvider)
Metrics
Reference-based (requires ideal answers in the dataset):
from aevyra_verdict import ExactMatch, BleuScore, RougeScore ExactMatch() # case-insensitive by default ExactMatch(case_sensitive=True) BleuScore(max_ngram=4) RougeScore(variant="rougeL") # also "rouge1", "rouge2"
Using these on a dataset without ideal answers raises a ValueError upfront — see Label-free evaluation below.
LLM-as-judge (works with or without ideal):
from aevyra_verdict import LLMJudge from aevyra_verdict.providers import get_provider judge = get_provider("openai", "gpt-5.4") LLMJudge(judge_provider=judge) LLMJudge(judge_provider=judge, criteria="Focus only on factual accuracy.")
The judge scores on a 1–5 scale (normalized to 0.0–1.0) and returns its reasoning in ScoreResult.reasoning.
Score across multiple dimensions in a single API call:
LLMJudge( judge_provider=judge, dimensions=["clarity", "accuracy", "conciseness"], ) # result.score → mean across all dimensions (0.0–1.0) # result.sub_scores → {"clarity": 0.8, "accuracy": 0.6, "conciseness": 1.0}
Custom metrics:
from aevyra_verdict import CustomMetric def word_count_score(response, ideal=None, **kwargs): return min(len(response.split()) / 100, 1.0) CustomMetric("word_count", word_count_score)
Custom functions return either a float or a dict with at least a "score" key
(optionally "reasoning" and any other details).
Label-free evaluation
When you have no reference answers, use LLMJudge (or a CustomMetric) instead of reference-based metrics. The runner validates this upfront and gives a clear error if you accidentally pair a label-free dataset with ExactMatch, BleuScore, or RougeScore.
# Dataset with no ideal answers dataset = Dataset.from_jsonl("questions.jsonl") print(dataset.has_ideals()) # False judge = get_provider("openai", "gpt-5.4") runner = EvalRunner() runner.add_provider("openai", "gpt-5.4-nano") runner.add_metric(LLMJudge(judge_provider=judge)) results = runner.run(dataset) # works fine — no labels needed
See examples/label_free_eval.py for a complete working example.
CLI
After pip install -e ., the aevyra-verdict command is available.
Inspect a dataset
Preview a dataset before running — shows sample count, whether ideals are present, and the first sample. No API calls made.
aevyra-verdict inspect examples/sample_data.jsonl
Check configured providers
List all available providers and whether their API keys are set:
aevyra-verdict providers
Specifying models
Pass --model (or -m) once per model, in provider/model format:
aevyra-verdict run examples/sample_data.jsonl \ -m openai/gpt-5.4-nano \ -m qwen/qwen3.5-9b \ -m google/gemini-2.0-flash
For more than a couple of models, or when you want to reuse a configuration, use a config file instead:
aevyra-verdict run examples/sample_data.jsonl --config models.yaml
The config file supports JSON, YAML, and TOML. Each model entry takes provider and model, with optional label, api_key, and base_url:
# models.yaml models: - provider: openai model: gpt-5.4-nano label: gpt-5.4-nano - provider: openrouter model: qwen/qwen3.5-9b label: qwen3.5-9b # Local vLLM instance — uses the OpenAI-compatible API - provider: openai model: meta-llama/Llama-3.1-8B-Instruct base_url: http://localhost:8000/v1 api_key: "none" # pragma: allowlist secret label: llama-local
Start a local vLLM server with: vllm serve meta-llama/Llama-3.1-8B-Instruct
Specifying metrics
Use --metric for built-in options (rouge, bleu, exact) and repeat for multiple:
aevyra-verdict run examples/sample_data.jsonl -m openai/gpt-5.4-nano --metric rouge --metric bleu
Add an LLM-as-judge with --judge:
aevyra-verdict run examples/sample_data.jsonl -m openai/gpt-5.4-nano --judge openai/gpt-5.4
To customise the judge's evaluation criteria, pass a prompt template file. The recommended format is .md since judge prompts tend to have structure. Use {criteria}, {conversation}, {response}, and {ideal_section} as placeholders:
aevyra-verdict run examples/sample_data.jsonl -m openai/gpt-5.4-nano \ --judge openai/gpt-5.4 \ --judge-prompt examples/judge_prompt.md
examples/judge_prompt.md is a copy of the default template — a good starting point.
To use a custom Python scoring function, point at a file and name the function:
aevyra-verdict run examples/sample_data.jsonl -m openai/gpt-5.4-nano \ --custom-metric examples/custom_metrics.py:brevity_score \ --custom-metric examples/custom_metrics.py:contains_code
The function receives (response, ideal=None, messages=None) and returns either a float (0.0–1.0) or a dict with a "score" key and optional "reasoning". See examples/custom_metrics.py for three working examples.
Save results to JSON with -o:
aevyra-verdict run examples/sample_data.jsonl --config models.yaml -o results.json
Results
results = runner.run(dataset) # Formatted comparison table print(results.compare("rouge_rougeL")) # Summary dict results.summary() # Pandas DataFrame df = results.to_dataframe() # Export to JSON results.to_json("eval_results.json")
Configuration
from aevyra_verdict.runner import RunConfig config = RunConfig( temperature=0.0, # deterministic by default max_tokens=1024, # Concurrency max_workers=10, # concurrent requests per model max_model_workers=4, # models evaluated concurrently # Retries and rate-limit handling num_retries=4, # attempts after the first failure retry_base_delay=1.0, # seconds before the first retry (doubles each attempt) retry_max_delay=60.0, # backoff cap in seconds retry_jitter=0.25, # ±25% random jitter to avoid thundering-herd retries ) runner = EvalRunner(config=config)
Rate-limit errors (HTTP 429 / RateLimitError) always sleep through the backoff
before retrying. Auth and bad-request errors are surfaced immediately — no point
retrying a 401. If you're consistently hitting rate limits, the first thing to try
is lowering max_workers.
Contributing
Bug reports and PRs are welcome. Open an issue first for anything larger than a bug fix.
Adding a provider — subclass Provider in src/aevyra_verdict/providers/, implement
complete(), and register it with register_provider(). See openai_provider.py as the
reference implementation.
Adding a metric — subclass Metric in src/aevyra_verdict/metrics/, implement
score(), and add it to the exports in metrics/__init__.py. If your metric requires
a reference answer, set requires_ideal = True on the class — the runner will then
raise a clear error when it's used on a label-free dataset. See reference.py for
reference-based metrics and judge.py for LLM-as-judge.
License
Apache 2.0