A single number tells you nothing. llm-eval-kit tells you why a response is good or bad — and how to fix it.

$ llm-eval refine "Propose un plan stratégique pour une startup IA en France en 2026."

╭─ Iteration 1 ──────────────────────────────────────────────────────╮
│  ⠹ Reasoning   ████████████░░░░░░░░  0.62                          │
│  ⠹ Factual     ███████████████░░░░░  0.78                          │
│  ⠹ Coherence   ██████████░░░░░░░░░░  0.55  ← weakest               │
│  ⠹ Safety      ████████████████████  0.99                          │
│           Overall: 0.74  →  refining...                            │
╰────────────────────────────────────────────────────────────────────╯
╭─ Iteration 2 ──────────────────────────────────────────────────────╮
│  ✓ Reasoning   ████████████████░░░░  0.84  (+0.22)                 │
│  ✓ Factual     ████████████████░░░░  0.83  (+0.05)                 │
│  ✓ Coherence   █████████████████░░░  0.87  (+0.32)                 │
│  ✓ Safety      ████████████████████  0.99  ( =  )                  │
│           Overall: 0.88  →  ✅ converged                           │
╰────────────────────────────────────────────────────────────────────╯

✓ Strengths:    Clear value-prop, sound market sizing
✗ Weaknesses:   Missing GTM timeline, vague pricing
→ Recommend:    Add 6-month milestones, propose 3 pricing tiers

uv pip install llm-eval-kit

pip install llm-eval-kit

uvx --from llm-eval-kit llm-eval evaluate "Explain quantum entanglement to a child."

git clone https://github.com/benmeryem-tech/llm-eval-kit.git
cd llm-eval-kit
make install-dev

pip install "llm-eval-kit[viz]"      # plotly + pandas + jinja2 (HTML dashboards)
pip install "llm-eval-kit[docs]"     # mkdocs-material
pip install "llm-eval-kit[logging]"  # structlog
pip install "llm-eval-kit[all]"      # everything

cp .env.example .env
echo "ANTHROPIC_API_KEY=sk-ant-..." >> .env

from llm_eval_kit import Evaluator, global_weighted_score, explain_score

evaluator = Evaluator(model="claude-3-5-sonnet-20241022")
result = evaluator.evaluate("Propose a strategic plan for an AI startup in France in 2026.")

score = global_weighted_score(result)
print(f"Overall: {score.overall_score:.2f}")
for dim, s in score.detailed_scores.items():
    print(f"  • {dim:<12} {s:.2f}")

explanation = explain_score(result)
print(f"\n✓ {explanation.strengths}")
print(f"✗ {explanation.weaknesses}")
print(f"→ {explanation.recommendations}")

from llm_eval_kit import SelfRefiner

refiner = SelfRefiner(model="claude-3-5-sonnet-20241022")
refined = refiner.refine(prompt, iterations=3)

print(f"Iterations:    {refined.iterations}")
print(f"Improvements:  {refined.improvement_count}")
print(f"Converged:     {refined.converged}")
print(f"Final score:   {refined.final.overall_score:.3f}")

llm-eval evaluate "Explain CRISPR in 3 sentences." --output report.html --format html
llm-eval refine "Write a haiku about recursion." --iterations 3 -v
llm-eval --version

from llm_eval_kit.critique.base import BaseCritic
from llm_eval_kit.core.types import CritiqueDimension
from llm_eval_kit import Evaluator

class ConcisenessCritic(BaseCritic):
    """Penalize verbose responses."""
    dimension = CritiqueDimension.COHERENCE

    def _build_prompt(self, prompt: str, response: str) -> str:
        return f"""Score this response on conciseness (0-1).
PROMPT: {prompt}
RESPONSE: {response}
Return JSON: {{"score": <float>, "rationale": "...", "issues": [...], "suggestions": [...]}}"""

evaluator = Evaluator(critics=[ConcisenessCritic()])
result = evaluator.evaluate("Explain recursion in one sentence.")

from llm_eval_kit.benchmarks import MultiModelBenchmark

bench = MultiModelBenchmark(
    models=["claude-3-5-sonnet-20241022", "claude-3-5-haiku-20241022"],
    dataset="reasoning_mini",
)
report = bench.run()
bench.export_html("dashboard.html")
bench.export_json("results.json")

for model, scores in report.summary.items():
    print(f"{model}: overall = {scores['overall']:.3f}")

from llm_eval_kit.critique import CORE_CRITICS, FULL_CRITICS, CREATIVE_CRITICS, SAFETY_CRITICS
evaluator = Evaluator(critics=FULL_CRITICS)

src/llm_eval_kit/
├── evaluator/          # Main orchestrator (sync + async)
├── refinement.py       # Self-refinement loop
├── scoring/            # Weighted multi-criteria scoring + registry
├── explainability/     # Strengths · weaknesses · recommendations
├── report/             # JSON · Markdown · HTML reports
├── providers/          # Anthropic · Mock · ProviderRegistry
├── critique/           # 8 critics + BaseCritic plugin API
├── benchmarks/         # Multi-model async runner + datasets
├── tracking/           # EvaluationStore (persisted runs)
├── config/             # Pydantic settings (.env)
├── core/               # Types · exceptions · logging
└── cli.py              # `llm-eval` entry point

from llm_eval_kit.providers import ProviderRegistry, AnthropicProvider, MockProvider
ProviderRegistry.register("my-llm", MyCustomProvider)

from llm_eval_kit import ScorerRegistry, WeightedScorer
ScorerRegistry.register("strict", WeightedScorer(weights={"safety": 0.6, "factual": 0.4}))

from llm_eval_kit import EvaluationStore
store = EvaluationStore("runs.db")
store.save(result)
history = store.query(model="claude-3-5-sonnet-20241022", min_score=0.8)

git clone https://github.com/benmeryem-tech/llm-eval-kit.git
cd llm-eval-kit
make install-dev
pre-commit install
make test
make lint
make format

If llm-eval-kit saved you time, please ⭐ the repo.

_{It's the single fastest way to support an indie open-source maintainer.}

⬆ Back to top

</di