What is nexa-gauge?

nexa-gauge is a Python package and command-line toolkit for evaluating generated outputs from LLM, RAG, and agentic systems. It replaces ad-hoc manual checks with a typed evaluation graph that can estimate cost, execute only the required nodes, reuse cached artifacts, and emit structured per-case reports.

It is designed for prompt iteration, benchmark runs, regression testing, release gates, and production evaluation workflows where teams need measurable quality and safety signals.

Core evaluation coverage includes:

• Relevance - measures whether generated claims answer the user question.
• Grounding - checks whether generated claims are supported by supplied context.
• Red team scoring - evaluates safety and risk behavior with configurable rubrics.
• GEval / LLM-as-a-judge - scores outputs against explicit criteria or evaluation steps.
• Reference metrics - computes overlap-based metrics against known reference answers.

Quickstart

Install the package from PyPI:

pip install nexa-gauge

Install optional Hugging Face dataset support:

pip install "nexa-gauge[huggingface]"

Set your provider key:

export OPENAI_API_KEY="<your-key>"

Estimate cost before running billable evaluation work:

nexagauge estimate eval --input sample.json --limit 10

Run the full evaluation graph and write per-case reports:

nexagauge run eval --input sample.json --limit 10 --output-dir ./report

Core Capabilities

• Graph-based evaluation pipeline - predictable node topology for scanning, chunking, claim extraction, metric execution, aggregation, and reporting.
• Estimate-first execution - preview uncached eligible cost before making LLM-backed calls.
• Cache-aware runs - avoid duplicate LLM spend and recomputation when inputs, prompts, and model routes are unchanged.
• Structured JSON reports - write per-case report files for CI, dashboards, notebooks, and downstream analytics.
• Per-node model routing - configure global models, node-specific models, fallback models, and temperatures.
• Scalable CLI execution - tune case-level workers, in-flight cases, and global LLM concurrency.
• Local and hosted datasets - evaluate JSON, JSONL, CSV, text files, and Hugging Face datasets.

Evaluation Pipeline

nexa-gauge runs evaluations through a deterministic node graph. Each target executes only its required upstream dependencies.

Typical execution paths:

grounding: scan -> chunk -> refiner -> claims -> grounding
relevance: scan -> chunk -> refiner -> claims -> relevance
geval:     scan -> geval_steps -> geval
eval:      full graph execution and aggregate metric summary

For a full architecture diagram, see docs/architecture.md.

CLI Usage

The CLI entry point is nexagauge.

nexagauge --help
nexagauge run --help
nexagauge estimate --help
nexagauge cache --help

Primary commands:

Common examples:

# Estimate full evaluation cost for a dataset slice
nexagauge estimate eval --input sample.json --limit 100

# Run full evaluation and write JSON reports
nexagauge run eval --input sample.json --limit 100 --output-dir ./report

# Run full evaluation with explicit chunk/refiner strategies
nexagauge run eval --input sample.json --limit 100 --chunker semchunk --refiner mmr --refiner-top-k 3

# Run only the grounding metric branch
nexagauge run grounding --input sample.json --limit 25

# Preview cache cleanup
nexagauge cache delete --dry-run

Common flags:

See docs/cli-code-flow.md and the hosted CLI documentation for deeper usage details.

Input Formats

Use --input with local files or hosted datasets.

Canonical record fields include:

Common aliases such as response, answer, output, completion, query, prompt, ground_truth, and label are normalized during scanning.

Metrics

nexa-gauge combines deterministic metrics with LLM-as-a-judge evaluation.

GEval is split into two phases:

1. geval_steps resolves reusable evaluation steps from criteria or accepts provided steps.
2. geval scores each case against those resolved steps and selected input fields.

This design makes rubric-based evaluation repeatable and cache-friendly across datasets.

Caching and Cost Estimation

Cost control is a first-class part of the runtime.

# Preview uncached work before execution
nexagauge estimate eval --input sample.json --limit 50

# Reuse cache during normal runs
nexagauge run eval --input sample.json --limit 50 --output-dir ./report

# Ignore cache reads but still write fresh outputs
nexagauge run eval --input sample.json --limit 50 --force

# Disable cache reads and writes for debugging
nexagauge run eval --input sample.json --limit 50 --no-cache

The cache is deterministic and route-aware. Inputs, evaluation criteria, model routing, prompt versions, parser versions, and relevant upstream artifacts are included in cache keys so stale outputs are not reused across incompatible runs.

For run, cache location can be controlled with:

export NEXAGAUGE_CACHE_DIR="./.nexagauge-cache"

Inspect the active cache root:

nexagauge cache dir

Clear cached node outputs:

nexagauge cache delete --dry-run
nexagauge cache delete --yes

Configuration

For local development or repeatable runs, copy the environment template:

cp .env.example .env

Minimum configuration for OpenAI-backed runs:

OPENAI_API_KEY=<your-key>
LLM_MODEL=gpt-4o-mini

Supported per-node overrides follow this pattern:

LLM_CLAIMS_MODEL=openai/gpt-4o-mini
LLM_CLAIMS_FALLBACK_MODEL=openai/gpt-4o
LLM_GROUNDING_TEMPERATURE=0.0

Runtime overrides can also be passed through the CLI:

nexagauge run eval \
  --input sample.json \
  --llm-model openai/gpt-4o-mini \
  --llm-model grounding=openai/gpt-4o \
  --llm-fallback openai/gpt-4o

Development

Clone the repository and install it from source:

git clone https://github.com/harnexa/nexa-gauge.git
cd nexa-gauge
pip install -e .

Contributor workflow with uv:

uv sync
make lint
make test
make ci

Build distributions:

uv build

Expected artifacts:

dist/nexa_gauge-<version>-py3-none-any.whl
dist/nexa_gauge-<version>.tar.gz

Releases use release-please. Conventional Commit titles such as feat:, fix:, docs:, deps:, and chore: are recommended for cleaner generated release notes, but they are not required by CI.

Documentation

• Hosted documentation: harnexa.dev/nexa-gauge/docs/introduction
• Local getting started guide: docs/get-started.md
• Architecture: docs/architecture.md
• CLI code flow: docs/cli-code-flow.md
• Product summary: docs/PRODUCT_SUMMARY.md

Project Standards

• License: MIT
• Security policy: SECURITY.md
• Contributing guide: CONTRIBUTING.md
• Code of conduct: CODE_OF_CONDUCT.md

License

Distributed under the MIT License. See LICENSE for details.