GEDD - A Systematic Evidence Driven LLM As a Judge Framework

GEDD is a Systematic Evidence Driven LLM As a Judge Framework for AI agents.

It is an annotation-first workflow for turning domain-owner review of AI agent behavior into release gates engineering can run.

The web app gives product managers, domain experts, and ML engineers one shared path:

1. Define the agent and the work it is supposed to do.
2. Collect or load representative queries and responses.
3. Review the responses in a task-shaped workbench.
4. Name failures in the domain owner's vocabulary.
5. Convert the observed failures into an LLM-as-a-judge prompt.
6. Export a validated handoff for CI, MLflow, and model regression work.

The current first-run experience ships with two 50-query PM workbench demos: an AAA game localization session and an AWS cloud GDPR auditor session. They show how a domain owner can move from raw agent traces to open codes, root-cause patterns, saturation evidence, a judge prompt, and an ML engineer implementation queue.

The longer methodology essay is in METHODOLOGY.md. This README is the practical product and engineering guide.

What GEDD Produces

GEDD is not a generic model leaderboard. It is a way to preserve expert judgment and make it executable.

Quick Start

Start the web app:

cd grounded-evals
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
grounded-evals serve --host 127.0.0.1 --port 8080

Open http://127.0.0.1:8080.

No Codex skill or plugin is required.

Local runs start in guest mode unless ADMIN_PASSWORD or Cognito environment variables are configured. If port 8080 is busy, use --port 8081.

For the fastest product tour, use one of the seeded 50-query demos. They do not require model calls:

1. Open Home or Demos.
2. Click Load 50-query localization demo or Load 50-query AWS Cloud GDPR demo.
3. Open PM Workbench to review the labeled traces, failure codes, memos, and saturation state.
4. Open Judge to inspect or revise the generated judge prompt.
5. Open Report to review release readiness and download the ML engineer handoff.

To reset after loading a demo, use the top-right refresh action. Confirm Start Fresh to clear the loaded project data while keeping the current login session.

Current Web App

grounded-evals serve runs a NiceGUI app with a short primary navigation:

The Demos page remains available for starter data. It is not the main workflow. Demos are seed sessions that help teams understand the annotation loop before they bring their own traces.

The 50-Query Localization Demo

The main demo is a synthetic but complete localization QA session for an AAA game agent called LocaleGate.

It includes:

Example failure codes in the demo include:

Those labels are the point of the workflow. The judge is not asked to score generic helpfulness first. It is asked to enforce the domain owner's observed release blockers.

The 50-Query AWS Cloud GDPR Demo

The second main workbench demo is a synthetic AWS cloud GDPR audit session for CloudAuditGate.

It includes 50 golden queries covering S3 and CloudWatch retention, CloudTrail and centralized logging, Bedrock prompt reuse, Rekognition and high-risk review, DSAR and deletion handling across backups and data lakes, shared responsibility, cross-region transfers, and breach escalation from AWS security incidents. The output is the same PM-owned package as the localization demo: annotations, open codes, axial coding, saturation evidence, and an audit-ready judge prompt.

The AWS Cloud GDPR demo uses plain-language tags on purpose, for example Data Used For The Wrong Job, Collecting Or Keeping Too Much Data, EU Data Moved The Wrong Way, and Trying To Work Around GDPR. The point is to make the GEDD loop easy to follow: annotate the failure in human language first, then turn that observed pattern into the judge gate.

Bring Your Own Agent

Use the app when you have a real or proposed agent and need review evidence before you automate evaluation.

If you already have production traces, use the app as an annotation surface rather than generating new responses. See Paste In Traces.

ML Engineer Handoff

The Report page contains an ML Engineer Handoff section. It is designed to be actionable, not a narrative status update.

It gives engineering:

Typical handoff commands:

cd grounded-evals

grounded-evals validate-session --session session.json
grounded-evals export --session session.json --format jsonl --output golden_dataset.jsonl
grounded-evals judge --session session.json --output judge_prompt.md
grounded-evals mlflow --session session.json --tracking-uri $MLFLOW_TRACKING_URI --run-eval

The expected engineering loop is:

1. Validate the session.
2. Create one failing regression case for each P0 queue item.
3. Patch the prompt, retrieval policy, tool policy, or runtime behavior.
4. Rerun the judge and review disagreements.
5. Promote the gate only after calibration is acceptable.

The default calibration target used in the handoff is kappa >= 0.80 before the judge blocks merges.

CLI Reference

The CLI supports the same workflow for repeatable runs, scripting, and CI.

Run command help from the package directory:

cd grounded-evals
grounded-evals --help
grounded-evals mlflow --help

Web App And CLI

GEDD ships as a web app and a CLI. No Codex skill or plugin is required.

Use the web app first unless you are automating an established workflow. The CLI is the right path for CI, MLflow, scripted exports, and headless checks.

Runtime And Provider Configuration

Local demo review does not require an LLM provider because the main localization demo is preloaded.

For custom agent work, configure one provider:

See SETUP.md for a full environment variable list, Bedrock model access notes, auth options, and AWS deployment setup.

Architecture

flowchart TD
    WEB["NiceGUI web app<br/>Home, Coach, Workbench, Judge, Report"]
    DEMO["Seeded demos<br/>50-query localization + domain scenarios"]
    SESSION["session.json<br/>agent, prompt, queries, labels, codebook"]
    REPORT["ML engineer handoff<br/>gates, artifacts, queue, commands"]
    CLI["grounded-evals CLI<br/>export, judge, validate, mlflow"]
    MLFLOW["MLflow / SageMaker MLflow<br/>datasets, scorers, runs"]
    CI["CI/CD gate<br/>regression checks"]
    RUNTIME["Agent runtime<br/>Bedrock, Anthropic, AgentCore"]

    DEMO --> WEB
    WEB --> SESSION
    WEB --> REPORT
    REPORT --> CLI
    SESSION --> CLI
    CLI --> MLFLOW
    MLFLOW --> CI
    CI --> RUNTIME
    RUNTIME --> WEB

Core paths:

Validation

Before committing app or workflow changes:

cd grounded-evals
PYTHONPATH=src pytest
PYTHONPATH=src python3 -m grounded_evals.cli --help

For local web smoke tests:

grounded-evals serve --host 127.0.0.1 --port 8080

for p in / /coding /demos /coach /judge /report /health; do
  curl -sS -o /dev/null -w "$p %{http_code}\n" "http://127.0.0.1:8080$p"
done

For README-only changes, git diff --check and stale-message scans are usually enough.

Additional Docs

License And Security

License: MIT-0. See LICENSE.

Security issue reporting: see CONTRIBUTING.md.