reasoning-core

Stop the agent vibecoding files outside its plan. Save your AI tokens. 100% locally.

TL;DR

git clone https://github.com/jakubkrzysztofsikora/reasoning-core.git
cd reasoning-core && python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt && huggingface-cli download state-spaces/mamba-130m-hf
direnv allow .
bash scripts/install-supervisor-launchagent.sh   # macOS — KeepAlive sidecar
# or, ad-hoc, no launchd:
# bash scripts/start-sidecar.sh
export PATH="$PWD/bin:$PATH"                     # `rc` admin shim
claude   # every Edit/Write Claude proposes is now scored before it lands

Also works headless on Gemini CLI, GitHub Copilot CLI, and Mistral Vibe CLI — same sidecar, per-host install scripts (§5c–5e).

After that, every change Claude proposes goes through a structural-regression scorer. The gate ships in shadow mode by default (RC_SHADOW_MODE=1) — decisions are logged to the audit trail without enforcing, so you can observe what would be blocked on your codebase before flipping it on. Repo-scoped via direnv — leaves every other folder untouched.

Table of contents

• What you get out of the box
  • More details
  • Where it doesn't help
• Why this exists
• The solution: System 1 + System 2
• Run it locally (6 steps, no global side-effects)
• Multi-CLI support — Gemini / Copilot / Vibe
• How it works under the hood
• Hook layers
• CLI
• Supervisor & launchd (macOS)
• Shadow mode & kill switches
• Evaluation harness
• Benchmarks — iteration 1 (draft)
• Scoring math
• Configuration
• Usage from code
• Project layout
• FAQ / troubleshooting
• Testing
• Roadmap
• Contributing
• Acknowledgements + License

What you get out of the box

• Up to ~29% fewer tokens per task. On the PR-review task in our 8-task eval, the sidecar pulled 724k cache-read tokens vs 1.02M for vanilla Claude Code — a 29% saving on that single task. Auth-abandonment task came in at 27% lower. At Anthropic's public cache-read price ($0.30/MTok), if you save ~300k cache-read tokens per task that's ~$0.09/task. 100 tasks/month ≈ $9 saved. 1,000 ≈ $90. 10,000 ≈ $900. Benchmark tests averaged across all 8 tasks saved 8.2%, while being more token-effecient on every task.

• The agent doesn't go vibecoding off-plan or inventing patterns your repo already has. Two measured wins drive this: the agent stays inside the files it promised to touch (+0.23 on a 1–5 scale, 4.09 → 4.31), and plans use your repo's existing helpers and naming conventions instead of inventing fresh ones (+0.43, 3.71 → 4.14). In practice: less PR-review thrash, less rework, fewer "no, use the existing util" loops with the agent.

• Plans you don't have to keep reviewing and amending. Plan quality went from 3.62 → 3.94 on the same 1–5 scale. You get a structured, sound plan on the first run.

• Privacy: 100% local, your code stays on your laptop. The scoring model (130M params, ~200MB RAM) and all the planning/grounding hooks run next to your claude CLI process. Nothing extra leaves your machine compared to plain claude — no telemetry, no cloud relay, no third-party service inspecting your diffs. Your code is processed exactly where it was already going (Anthropic) and nowhere else.

More details

8 real engineering tasks. 3 runs each. 2 setups (vanilla claude vs claude + sidecar). 3 independent reviewer models from 3 different vendors graded every plan and every implementation, blind.

Where it doesn't help

• It costs you ~98 seconds per run. The sidecar plans before the agent edits. If you live or die by raw turnaround, vanilla claude is faster.
• Code legibility was a tie. Both setups produce equally readable code. The sidecar doesn't make your code prettier — it makes the agent comply with your standards and stay on-plan.
• Eval was 1 codebase, 8 tasks. Numbers may shift on yours.

Why this exists

Where LLMs shine

LLMs are extraordinary at token-level pattern completion. Function signature + docstring → plausible body. Stack trace → suggested fix. Dataset + goal → boilerplate. Frontier models match or exceed mid-level engineers on bounded coding tasks where the answer is a local rewrite.

Where LLMs fail

Drop the same model into a 50k-LOC codebase and quality collapses. LLMs are bad at:

• Architectural invariants — does this preserve the layer boundary? The model has no concept of "layer", just tokens.
• Cross-file consequences — a 1-line edit fan-outs to 30 callers; the model only saw 1 file.
• Cyclomatic envelopes — adding a 4th conditional to an already-8-branch function makes it untestable; the model sees "another if".
• Coupling drift — each new import creeps the module further from its single responsibility, with no penalty in the loss.
• Coherence vs the rest of the project — a new helper introducing a different idiom from the codebase is "novel", but novelty is sometimes a bug not a feature.
• Plan→implementation alignment — plan said 4 phases, only 2 landed; plan said touch auth/, diff also rewrote payments/. The model doesn't audit itself.

Why they fail (the structural blind spot)

LLMs reason over token streams. They don't have a graph of who calls whom, a tree of which scope nests where, or a metric for how "different" your file is from the project's attractor. Those properties live in the structure — the AST, the call graph, the embedding manifold of the repo as a whole.

Long-context windows (1M tokens) help but don't solve. The model still has to infer structure from tokens at every inference, with no persistence. Reasoning over structure is a different kind of computation: graph diffs, embedding distances, dimensionality. LLMs aren't optimized for it; they do it badly compared to a small specialist.

Will scaling fix it?

Probably not, at least not soon.

• Compute scaling — 2024-25 long-context literature shows degraded consistent cross-document reasoning even at 1M+ tokens. Bottleneck is architectural, not budget.
• Tool scaling — agents calling structural tools (linters, type checkers, AST search) outperform ones that don't. The field is going there. But tool calls are stateless per question; they don't accumulate repo-level reasoning.
• Specialized models — small models trained for code structure (CodeBERT-style retrievers, code-aware embedders) keep outperforming much larger general models on code-similarity. A small specialist beats a large generalist for this sub-problem.

So the forecast: LLMs keep getting better at the linguistic surface — intent decoding, emission, prose. They will not spontaneously develop reliable structural reasoning. Hybrid systems where the LLM defers structural decisions to a specialist will be the durable architecture.

That's what this is.

The solution: System 1 + System 2

Loose nod to Kahneman: System 1 = fast, linguistic, intuitive (LLM); System 2 = slow, structural, deliberate (the SSM scorer).

Claude proposes an edit
        │
        ▼
┌─────────────────────────────┐
│ pre_edit_guard.py (hook)    │  reads disk + new_string,
│                             │  reconstructs (before, after),
└──────────────┬──────────────┘  posts to the sidecar
               │
               ▼
┌─────────────────────────────────────────────┐
│ Sidecar (FastAPI, 127.0.0.1:8765)           │
│   • Tree-sitter parse → AST + call graph    │
│   • Mamba 130M forward → pooled embedding   │
│   • 8-dim risk vector (delta semantics)     │
│   • Per-kind thresholds (source/test/plan)  │
│   • Cold-start aware (new files don't lie)  │
└──────────────┬──────────────────────────────┘
               │ ImpactReport JSON
               ▼
┌─────────────────────────────┐
│ Hook decides:               │
│   regression? → exit 2,     │
│     stderr block w/ top-3   │
│     repair hints            │
│   safe? → exit 0, edit      │
│     proceeds                │
└─────────────────────────────┘

Concrete example. Claude proposes:

-def normalize(items):
-    if not items:
-        return []
-    return [x.lower() for x in items]
+def normalize(items):
+    return [x.lower() for x in items]

Sidecar response (abbreviated):

{
  "architectural_impact_score": 0.31,
  "coherence_delta": 0.44,
  "file_kind": "source_code",
  "risk_vector": {"novelty": 0.94, "churn": 0.02, "...": 0.0},
  "regression_detected": true
}

Hook blocks with stderr listing top-3 contributors + repair hints. Claude re-reads, revises, retries. (Today: one-shot allow/block; iterative-loop on the roadmap.)

See docs/ARCHITECTURE.md for the deep-dive and docs/HARDENING.md for the threat model.

Run it locally (6 steps, no global side-effects)

The recommended setup is repo-scoped: leaves every other repo and Claude session untouched. Promote to global only after you're convinced it earns its keep.

1. Clone + venv

git clone https://github.com/jakubkrzysztofsikora/reasoning-core.git
cd reasoning-core
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

2. One-time Mamba checkpoint cache (~250 MB)

huggingface-cli download state-spaces/mamba-130m-hf

3. Boot the sidecar

bash scripts/start-sidecar.sh
# CPU first run: ~30s to load Mamba weights
curl -fsS http://127.0.0.1:8765/health | jq .model_loaded   # → true

4. Activate direnv for repo-scoped env

The repo ships an .envrc that loads venv, sidecar tuning, hook policy posture, HuggingFace cache pointer (shared with sibling repos at $HOME/.cache/huggingface so weights aren't downloaded twice), and a Cato VPN-aware CA bundle only when you cd into this folder. Other repos see none of it.

brew install direnv                          # if not installed
echo 'eval "$(direnv hook zsh)"' >> ~/.zshrc # or bash equivalent
cd ~/Repos/personal/reasoning-core
direnv allow .
export PATH="$PWD/bin:$PATH"                 # so `rc` shim resolves

Secrets / personal toggles → .envrc.local (gitignored, sourced last).

4b. (Optional, non–Apple-silicon) Use Scaleway-hosted generative critic

On Linux / CI / Intel Mac the local mlx_lm.server path is unavailable. Point the generative critic at Scaleway's hosted OpenAI-compatible API instead (Bearer auth via RC_GEN_API_KEY or SCALEWAY_API_KEY):

export RC_REASONER_BACKEND=remote
export RC_GEN_URL=https://api.scaleway.ai/v1/chat/completions
export RC_GEN_API_KEY=$(scw config get secret-key --profile circit)  # or your own key
export RC_GEN_MODEL=qwen3-coder-30b-a3b-instruct                     # or devstral-2-123b-instruct-2512
export RC_GEN_BUDGET_MS=15000

Apple-silicon users keep the local path — no changes needed.

5. Launch Claude from this folder

cd /path/to/reasoning-core
claude   # picks up .claude/settings.json — hooks active for THIS session only

Verify it's actually thinking:

curl -fsS http://127.0.0.1:8765/metrics | jq                              # score_calls, p50_ms, p95_ms
ls ~/.local/share/reasoning-core/events/$(date +%F)/ | head               # per-decision audit log
rc status                                                                 # sidecar health + posture

5b. Enable on another repo (3 commands)

You've already cloned reasoning-core and the sidecar is running. To gate one of your other repos with the same hooks, run from inside that repo:

export RC_REPO=$HOME/Repos/personal/reasoning-core   # or wherever you cloned it
bash $RC_REPO/scripts/enable-in-repo.sh              # writes .envrc + .claude/settings.local.json
direnv allow .                                       # load the env

That's it. claude from that repo now goes through reasoning-core. The script refuses to overwrite an existing .envrc or .claude/settings.local.json, and it only writes inside the cwd — nothing global is touched. Iter-3 levers (RC_BEST_EFFORT_SPEC, RC_PLAN_GROUNDING) are commented out in the generated .envrc; uncomment + direnv reload to opt in.

To revert: delete .envrc and .claude/settings.local.json from the repo.

5c. Enable for Gemini CLI

gemini hooks migrate makes Gemini's hook surface Claude-compatible by design. The same sidecar + adapter layer serves Gemini agents transparently — only the config file path differs.

export RC_REPO=$HOME/Repos/personal/reasoning-core
bash $RC_REPO/scripts/enable-in-repo-gemini.sh        # writes .gemini/settings.json + .gemini/skills/
direnv allow .
gemini --yolo "say hi"                                # --yolo bypasses MCP trust prompt

The install script renders .gemini/settings.json.template with <RC_REPO> substituted to an absolute path (per-machine; gitignored). The template ships event names that Claude already uses (PreToolUse, PostToolUse, SessionStart, UserPromptSubmit, PreCompact) — no rename table.

Verified against gemini v0.37.1.

5d. Enable for GitHub Copilot CLI

GitHub Copilot CLI v1.0.29 has no hook subcommand — runtime gating is the gate_edit MCP tool only, enforced by .copilot/copilot-instructions.md (always loaded). Post-turn audit retroactively flags missed calls (see eval/reconcile_session.py).

export RC_REPO=$HOME/Repos/personal/reasoning-core
bash $RC_REPO/scripts/enable-in-repo-copilot.sh       # merges hybrid-reasoner into ~/.copilot/mcp-config.json
copilot --allow-all-tools "say hi"                    # or env COPILOT_ALLOW_ALL=1

The script preserves any existing user MCP servers via atomic temp-rename with a timestamped backup. Tier-2 caveat: under context pressure the agent will sometimes skip gate_edit — for mission-critical work prefer Claude or Gemini, where the gate is a runtime hook.

5e. Enable for Mistral Vibe CLI

Vibe v2.9.4 ships only a post-agent-turn hook today. Like Copilot, the runtime gate is gate_edit MCP; instructions live in .vibe/AGENTS.md (Vibe-scoped, NOT repo-root, to avoid double-scoring on hosts that also read AGENTS.md).

export RC_REPO=$HOME/Repos/personal/reasoning-core
bash $RC_REPO/scripts/enable-in-repo-vibe.sh          # writes .vibe/config.toml + AGENTS.md, registers in trusted_folders
direnv allow .
vibe --prompt "say hi" --trust                        # --trust skips per-invocation trust prompt

For more on per-host gaps and the production caveats (tier-2 vs tier-1 gating, flock-on-macOS, direnv allow requirement, MCP startup races), see docs/CLI_PARITY.md.

6. (Optional) Promote globally — one path, every project

Repo-scoped hooks fire only when Claude runs from inside this folder. To get the same gating across every project on your machine, register the hooks once at the user level — no per-repo copy-paste, no .claude/settings.json in each project.

The hooks are designed to live in this single clone. Other projects reference them by absolute path via the RC_REPO env var, so a regex tweak or a hook fix here propagates everywhere on the next session.

Order matters — follow steps 0 → 5 in order. Sidecar must be daemonized BEFORE flipping S2_FAIL_CLOSED, otherwise every Edit on every project hard-blocks until the sidecar is up.

Step 0. Preflight — confirm the venv has the deps

The hooks import tree_sitter, transformers (HF Mamba port), fastapi, and a handful of other non-stdlib packages that live in this repo's requirements.txt. Hooks invoked via system python3 will ImportError on first fire and silently break every Claude session machine-wide. Confirm:

cd $RC_REPO
# Create venv + install runtime deps (this is the same install §1 ran for the
# in-repo session — you only need to do it once, but verifying here costs nothing).
test -x .venv/bin/python || python3 -m venv .venv
.venv/bin/pip install -r requirements.txt          # heavy deps live here, NOT in pyproject.toml
.venv/bin/python -c "import tree_sitter, transformers, fastapi; print('venv OK')"

If you previously did §1 (pip install -r requirements.txt), the venv is already ready — the verify line is enough.

Step 1. Pin the clone path in your shell rc (~/.zshrc or ~/.bashrc)

export RC_REPO="$HOME/Repos/personal/reasoning-core"   # wherever you cloned
export PATH="$RC_REPO/bin:$PATH"                        # so `rc` shim resolves anywhere

Reload: exec zsh (or source ~/.zshrc).

Step 2. Daemonize the sidecar

Always-on sidecar — no bash scripts/start-sidecar.sh in every shell.

macOS:

bash $RC_REPO/scripts/install-supervisor-launchagent.sh
launchctl list | grep reasoning-core      # → com.reasoning-core.supervisor visible
curl -fsS http://127.0.0.1:8765/health    # → {"status":"ok",...}

Linux (systemd user unit):

mkdir -p ~/.config/systemd/user
cat > ~/.config/systemd/user/reasoning-core-sidecar.service <<EOF
[Unit]
Description=reasoning-core SSM sidecar
[Service]
Type=simple
ExecStart=$RC_REPO/.venv/bin/python -m src.s2_core
WorkingDirectory=$RC_REPO
Restart=on-failure
RestartSec=5
[Install]
WantedBy=default.target
EOF
systemctl --user daemon-reload
systemctl --user enable --now reasoning-core-sidecar.service
curl -fsS http://127.0.0.1:8765/health    # → {"status":"ok",...}

Step 3. Register the hooks at the user level

Save the snippet below to ~/.claude/settings.json (merge into your existing file if present — append to the matching event arrays, do NOT replace the whole hooks block or you lose any other hooks you had).

Notes on the snippet:

• The .venv/bin/python prefix is load-bearing — system python3 lacks the deps.
• ${RC_REPO:-fallback} is POSIX parameter expansion, evaluated by Claude Code's shell-invoked hook command. The fallback path is for safety; replace with your real clone path. One-liner if you set $RC_REPO already:

  sed -i '' "s|/Users/you/Repos/personal/reasoning-core|$RC_REPO|g" ~/.claude/settings.json   # macOS
  sed -i    "s|/Users/you/Repos/personal/reasoning-core|$RC_REPO|g" ~/.claude/settings.json   # Linux

• session_resume_inject.py is registered on BOTH SessionStart AND UserPromptSubmit — that's the in-repo .claude/settings.json wiring; both events are needed so resume-context fires consistently.

Step 4. Set conservative env defaults globally

Append to your shell rc. Critical: leave S2_FAIL_CLOSED=0 until you've verified the daemon stays up across reboots — fail-closed globally + sidecar crash = every Edit on every project hard-blocks.

# ~/.zshrc
export RC_SHADOW_MODE=1            # log-only by default; flip to 0 once calibrated
export RC_LANG_LOCK=0               # off by default — repos vary; calibrate per-repo via project .envrc
export RC_PLAN_BLOCK=0              # plan-guard warn-only by default
export S2_FAIL_CLOSED=0             # fail-OPEN globally; flip to 1 only after the daemon proves stable
# Iter-3 levers stay project-scoped — DO NOT enable globally without registering
# session_start_best_effort.py first (see docs/iter3-levers.md):
# export RC_BEST_EFFORT_SPEC=1
# export RC_PLAN_GROUNDING=1

Per-repo .envrc files override these for projects where you want stricter posture (e.g. set S2_FAIL_CLOSED=1 and RC_PLAN_BLOCK=1 in .envrc for the repo where you've already calibrated the corpus).

Step 5. Verify global activation

Open Claude Code from any repo (not just $RC_REPO) and confirm hooks fire:

cd ~/some/other/project
claude
# … in another terminal …
ls ~/.local/share/reasoning-core/events/$(date +%F)/   # ← decisions from THIS project's session appear
rc status                                              # sidecar /health passes
jq -r '.project_dir' ~/.local/share/reasoning-core/events/$(date +%F)/*.jsonl | sort -u
# ← project_dir field shows the actual cwd of the session, not $RC_REPO

Iter-3 lever sanity (only if you opted into them in step 4):

jq 'select(.signal_source=="best_effort_spec" and .decision=="injected") | .overlay_sha' \
   ~/.local/share/reasoning-core/events/$(date +%F)/*.jsonl
# ← should print "c7080f353e94" (overlay v3) once per session

Notes & gotchas

• ~/.claude/settings.json + project-local .claude/settings.local.json MERGE ADDITIVELY on hook arrays. Each matcher/event entry is appended; matching entries do NOT override. If pre_edit_guard.py is registered both globally and in a project's settings.local.json, the hook runs twice per edit. Solution: pick one — global-only (recommended for shared hooks) or project-only (recommended for experimental ones). Don't double-register.
• Cato VPN / corporate TLS-MITM: the in-repo .envrc builds a certifi+Cato CA bundle (~/.cache/reasoning-core/ca-bundle.pem) and exports SSL_CERT_FILE / REQUESTS_CA_BUNDLE / CURL_CA_BUNDLE. If you go global on a corp-network laptop, copy that block (or the bundle path) into your shell rc — otherwise Scaleway / HF Hub TLS will fail with cert-chain errors.
• Iter-3 levers won't fire globally unless you ALSO register session_start_best_effort.py in the SessionStart array above. The repo .envrc has a _RC_LEVER_WARN self-check that catches env-set-but-hook-unregistered; global users don't run direnv on ~/, so the warn is silent. Either register the hook globally and document iter-3 enablement, or keep the levers project-scoped via per-repo .envrc. See docs/iter3-levers.md.
• Uninstall: remove the reasoning-core entries from ~/.claude/settings.json with jq rather than deleting the whole hooks block (you may have other hooks):

  jq 'del(.hooks.PreToolUse |= map(select(.hooks[0].command|test("reasoning-core")|not)))' \
     ~/.claude/settings.json > /tmp/s.json && mv /tmp/s.json ~/.claude/settings.json
  # repeat for PostToolUse, SessionStart, UserPromptSubmit, PreCompact, then
  launchctl unload ~/Library/LaunchAgents/com.reasoning-core.supervisor.plist  # macOS
  systemctl --user disable --now reasoning-core-sidecar.service                  # Linux

• Failure modes to know:
  • RC_REPO wrong/unset: hooks 404, every Edit fails. Symptom: stderr from Claude Code shows command not found or python3: can't open file. Fix: re-export or fix the fallback path.
  • Sidecar dead + S2_FAIL_CLOSED=1: every Edit hard-blocks. Symptom: [hybrid-reasoner] BLOCKED: sidecar unreachable. Fix: rc status, restart the daemon (step 2).
  • venv missing tree_sitter / transformers: every hook ImportErrors. Symptom: stderr noise from every Edit (ModuleNotFoundError: No module named 'tree_sitter'). Fix: cd $RC_REPO && .venv/bin/pip install -r requirements.txt.

How it works under the hood

1. Claude proposes an edit. The PreToolUse hook fires before the file is modified.
2. pre_edit_guard.py reads the hook payload from stdin, reconstructs the post-edit file (before from disk + apply old_string→new_string), POSTs {path, before_src, after_src} to the sidecar.
3. Tree-sitter parses both sides into ASTs. Code languages get a per-module call graph; data languages skip the graph but still get embedded.
4. Mamba SSM produces a pooled embedding for each side (130M params, hidden=768, CPU).
5. Risk vector (8 dims, all delta-semantics): cyclomatic, fan_in, fan_out, depth, churn, coupling, cohesion, novelty.
6. File-kind dispatch picks per-kind cd/ais/dim thresholds. Cold-start (empty before) zeros structural dims so new files only get gated on content novelty.
7. The hook blocks (exit 2) iff:
   • architectural_impact_score < ais_threshold[kind]
   • coherence_delta > cd_threshold[kind]
   • any risk dim > dim_ceiling[kind]
8. Block stderr surfaces top-3 risk contributors with repair hints. Retries within 120s trigger a "RETRY DETECTED" banner.
9. In shadow mode (default), steps 7–8 still execute but the hook always returns exit 0; the would-be decision is logged for offline review.

Hook layers

All nine wired in .claude/settings.json. Every fire emits an audit row.

Internal helpers (libraries, not hook entrypoints): _audit_rotation, _block_format, _kill_switches, _magic_comments, _mock_detector, _ood_detector, _plan_quality, _session_manifest, _shadow_mode.

CLI

Put bin/ on PATH (export PATH="$PWD/bin:$PATH") and use rc for diagnostics and single-shot bypasses:

rc --help is authoritative.

Supervisor & launchd (macOS)

The sidecar is a long-lived FastAPI process; if it crashes mid-session you lose scoring until you notice. The supervisor solves both problems.

bash scripts/install-supervisor-launchagent.sh
launchctl list | grep com.reasoning-core
tail -f /tmp/rc-sidecar-supervisor.log

• KeepAlive=true → relaunches on crash.
• RunAtLoad=true → starts on login.
• Uninstall: launchctl bootout gui/$UID/com.reasoning-core.supervisor then delete the plist from ~/Library/LaunchAgents/.

Linux equivalent: see docs/HARDENING.md.

Shadow mode & kill switches

The gate ships in shadow mode by default (RC_SHADOW_MODE=1 in .envrc). Decisions are computed and logged; the hook always returns exit 0. This lets you observe what the gate would have done on your codebase before flipping it on.

Promote to enforcement when ready:

echo 'export RC_SHADOW_MODE=0' >> .envrc.local
direnv reload

Escapes (in order of preference):

• Magic comment, single edit: prepend # rc:bypass-next (or // rc:bypass-next) to the file before the Edit Claude is about to fire.
• Single-shot, single command: rc bypass-next arms one bypass; the next guard fire consumes it.
• Single-shot, fresh session: RC_BYPASS_NEXT=1 claude ... — captured at session boot, consumed by the first guard fire.
• Per-path session-wide: RC_ALLOW_GUARD_EDIT=1 for guarded paths, RC_ALLOW_SUBAGENT_GUARD_EDIT=1 for subagent prompts naming them.
• Last resort: S2_FAIL_CLOSED=0 and kill the sidecar — fails open. Don't ship this; it nullifies the gate.

Every escape path emits an audit row tagged with the override mechanism so abuse is spottable later.

Evaluation harness

eval/ is the calibration + regression-test machine for the gate. See eval/README.md for full reference.

Smoke run:

python3 -m eval.run_suite --task fixtures/smoke --n 2
python3 -m eval.aggregate --runs eval/runs/smoke-001

Benchmarks — iteration 1 (draft)

Status: initial draft, n=1–3 per cell, single-judge. Iteration-2 failure-mode fixes are shipped (mock-detector P1 1724810, plan-quality CGS P2 ad2ea80, language-fingerprint lock P3 54c6e57, calibration concurrent with shadow P7 89820b0); the binomial sign-test acceptance harness wired in eval/stats.py::sign_test (6a921ce). Re-run against the same 8-task harness is scheduled, not yet executed — the numbers below are still iter-1 ground truth. Pre-registered acceptance: ≥7/8 task-mean wins for Setup B with ≥1.0 BARS gap and sign-test p ≤ 0.05. Full per-task per-judge tables live in ~/evals/2026-05-05_*/REPORT.md (gitignored — they include real Anthropic spend).

Setup. Two arms × eight tasks (T1, T2, T5, T7, T8, T9, E1, P0), randomized arm order per pair, fixed-seed correctness gate + rotated-seed flake check, BARS-rubric implementation- and plan-quality grades.

• Setup A (control): vanilla Claude Code, no hooks.
• Setup B (reasoning-core): the gate in this repo, S2_FAIL_CLOSED=1, shadow-mode off (a measurement configuration; default operator posture ships shadow-mode-on per .envrc).

Headline numbers (means across 8 tasks)

Suite totals: Setup A spent $91.50 / 533 866 tokens across all 8 tasks; Setup B spent $68.51 / 521 772 tokens. ~$23 / 25% saved at the suite level on this single-run draft.

Per-task verdicts

What this draft shows

• Money: Setup B is meaningfully cheaper on 6/8 tasks. The P0 outlier (B spent $21.58 vs A's $8.71) inflates B's mean tokens and partially erases the per-token savings; without P0, B's mean cost drops to ~$5.34 (−40% vs A).
• Wall clock: Setup B finishes ~23% faster on average. The gate is not free (p95 ~5 s/Edit on CPU Mamba); the speedup comes from B avoiding regression-rework loops.
• Quality: implementation-quality means are flat. B wins by decision rule (gates → impl_q → plan_q → cost), not by raw rubric points.
• Failures (informative):
  • T1 lost because the iter-1 build had no mock-detector — Claude shipped placeholder code, scored low. Iter-2 ships _mock_detector.py.
  • T9 lost because plan-time scoring measured plan-vs-plan novelty, not generic-vs-specific. Iter-2 ships the plan-quality CGS gate (_plan_quality.py, behind RC_PLAN_QUALITY=1).
  • E1 is a partial-win: B passed the correctness gate (locked 1/1) where A failed (locked 0/1), but the rubric grader marked B's diff lower because no language-convention enforcement existed in iter-1. Iter-2 ships RC_LANG_LOCK + post-batch language audit.

Caveats

• n=1 per (task, arm) cell on most tasks (T5/T7 group has n=3); CIs are wide. The iter-2 re-run targets n≥3 per cell with cross-family judges (Gemini + vibe) and a Krippendorff α inter-rater gate.
• Single-judge BARS grades; iter-2 adds the cross-family judge and per-grade contamination check (eval/contamination.py in the orchestrator harness).
• Decision rule is lexicographic (gates first); a single rubric point swing can flip a per-task verdict. Treat headline win-count as directional, not significant.

Iteration-2 acceptance criterion (pre-registered)

Setup B must pass the sign-test on 8 tasks (≥7/8 wins → p ≤ 0.035; 8/8 → p = 0.0039) with paired bootstrap 95% CI on suite-mean BARS impl-quality excluding 0. See thoughts/shared/plans/2026-05-06-iter2-100pct-eval-plan.md for the full v2 plan + the 19 reviewer corrections folded in.

Scoring math

8-dim risk vector (all delta semantics)

Coherence delta

coherence_delta = ||emb_after − emb_before||₂ / sqrt(hidden_size). Cold-start (empty before_src or <32 chars) sets it to 0.0.

Per-kind thresholds

Block fires iff ais < threshold[kind] OR cd > threshold[kind] OR any risk_dim > dim_ceiling[kind].

Configuration

Sidecar runtime

Source-code thresholds

Per-kind ceilings for test_code / plan_md / doc_md / config are not env-overridable yet — see _KIND_THRESHOLDS in src/s2_core.py. The three vars below control only the source_code kind.

Hook policy posture

Generative repair head

Calibration & supervisor (P7)

Bypass / kill switches

Audit log & state

Eval / calibration

Usage from code

from src.s2_core import score_change

before = "def f():\n    if not x: return\n    return x.lower()\n"
after  = "def f():\n    return x.lower()\n"
r = score_change("/repo/util.py", before, after)
print(r.architectural_impact_score, r.regression_detected, r.file_kind)

curl -fsS -X POST http://127.0.0.1:8765/score \
  -H 'content-type: application/json' \
  -d '{"path":"/repo/util.py","before_src":"...","after_src":"..."}' | jq

Project layout

reasoning-core/
├── README.md                       ← you are here
├── LICENSE
├── requirements.txt
├── pyproject.toml
├── .envrc                          ← repo-scoped env (direnv)
├── .claude/
│   ├── settings.json               ← 9 hook matchers + MCP server
│   └── skills/reasoning/SKILL.md
├── bin/
│   └── rc                          ← `python3 -m src.rc_cli` shim
├── launchd/
│   └── com.reasoning-core.supervisor.plist
├── src/
│   ├── ssm_backbone.py             ← Mamba loader, embed(), ast_to_tokens
│   ├── grammars.py                 ← Tree-sitter loader (12 languages)
│   ├── s2_core.py                  ← parsing, scoring, FastAPI sidecar
│   ├── mcp_reasoner.py             ← FastMCP bridge
│   ├── calibration.py              ← Mahalanobis + Ledoit-Wolf shrinkage + empirical-Bayes per-kind
│   ├── gen_client.py               ← Qwen / Scaleway generative client (Bearer auth)
│   ├── sidecar_supervisor.py       ← KeepAlive supervisor
│   ├── _supervisor_broker.py       ← cross-process broker (`/health` aggregator on RC_BROKER_PORT)
│   ├── _supervisor_env.py          ← env capture + restore
│   ├── _supervisor_recalibrate.py  ← watcher consuming `recalibrate.signal` for hot-refit (`5313498`)
│   ├── rc_cli.py                   ← admin / diagnostic CLI
│   └── hooks/
│       ├── pre_edit_guard.py
│       ├── pre_plan_guard.py
│       ├── pre_bash_guard.py
│       ├── pre_task_guard.py
│       ├── pre_compact_guard.py
│       ├── post_bash_revive.py
│       ├── post_batch_lang_audit.py
│       ├── session_start_manifest.py
│       ├── session_resume_inject.py
│       ├── audit_log.py            ← JSONL audit + retry detection
│       ├── _block_format.py        ← block message + repair hints
│       ├── _audit_rotation.py
│       ├── _kill_switches.py
│       ├── _magic_comments.py
│       ├── _mock_detector.py
│       ├── _ood_detector.py
│       ├── _plan_quality.py
│       ├── _session_manifest.py
│       ├── _shadow_mode.py
│       └── _calibration_gate.py    ← hot-path Mahalanobis gate (`b7f0517`)
├── scripts/
│   ├── start-sidecar.sh
│   ├── start-gen-sidecar.sh
│   ├── install-supervisor-launchagent.sh
│   ├── configure-scaleway.sh
│   └── test-prototype.sh
├── tests/
├── eval/
│   ├── README.md
│   ├── Dockerfile
│   ├── run_suite.py                ← top-level harness
│   ├── run_task.sh
│   ├── aggregate.py
│   ├── stats.py                    ← paired Wilcoxon
│   ├── metrics.py
│   ├── validate_embedder.py        ← embedder fitness test
│   ├── calibration_corpus.py       ← labeled corpus mining
│   ├── golden_set.py               ← regression suite
│   ├── recalibrate.py              ← Page-Hinkley monthly recal
│   ├── synthetic_drift.py
│   ├── build_grounding_pairs.py
│   ├── qwen_grounding_eval.py      ← Cohen κ ≥ 0.7 gate
│   ├── datasets/
│   │   ├── grounding_pairs.jsonl   ← 200 hand-labeled pairs
│   │   ├── swe_bench_verified_python_subset.json
│   │   └── refresh_subset.py
│   ├── prompts/system_prompt.txt
│   ├── fixtures/
│   ├── runs/
│   └── scripts/prefetch_mamba.sh
├── thoughts/shared/                ← research, plans, handoffs
└── docs/
    ├── ARCHITECTURE.md
    ├── HARDENING.md
    ├── EVAL_DESIGN.md
    ├── EVAL_RESULTS.md
    ├── VERIFICATION.md
    └── PLAN.md

FAQ / troubleshooting

Q: First time running, am I getting blocked? A: No. The gate ships in shadow mode (RC_SHADOW_MODE=1). Every Edit/Write is scored and the decision is logged to ~/.local/share/reasoning-core/events/, but the hook always returns exit 0. Promote to enforcement after a few sessions of observation by setting RC_SHADOW_MODE=0 in .envrc.local.

Q: The hook keeps blocking obviously-fine edits. A: Check top risk contributors in the block message. If a single dim sits at 1.00 on a tiny edit, restart the sidecar (bash scripts/start-sidecar.sh) — old processes can hold pre-refactor scoring code. If it persists, run rc status and rc explain, then open an issue with the audit row attached.

Q: Sidecar keeps dying mid-session. A: Install the launchd supervisor: bash scripts/install-supervisor-launchagent.sh. KeepAlive will relaunch it on crash and on login.

Q: I'm on a corporate VPN (Cato / Zscaler / etc.) and pip install / huggingface-cli fail with "self-signed certificate in certificate chain". A: direnv reload — the repo's .envrc builds ~/.cache/reasoning-core/ca-bundle.pem from certifi + your system Cato root and exports SSL_CERT_FILE / REQUESTS_CA_BUNDLE / CURL_CA_BUNDLE. macOS only today (uses security find-certificate); Linux users add their root to the bundle manually.

Q: Sidecar takes forever to start. A: First run downloads Mamba weights (~250 MB). Subsequent boots ~30s on CPU. Watch tail -f /tmp/reasoning-core-sidecar.log.

Q: How do I temporarily turn it off? A: cd out of the repo (direnv unloads, hooks vanish), or export S2_FAIL_CLOSED=0 and kill the sidecar (hooks fail-open).

Q: I want to edit the guard files themselves. A: Set RC_ALLOW_GUARD_EDIT=1 in the shell that started Claude, restart Claude, edit. The env is captured at session boot.

Q: Will it slow me down? A: p95 ~5s per Edit on CPU. Latency is in the Mamba forward pass; CUDA / MLX kernels would cut it to ~50ms. Tracked in roadmap.

Q: It blocked a legitimate refactor. How do I override? A: Either revise to address the top-3 risk contributors (recommended), or set RC_ALLOW_GUARD_EDIT=1 for guarded paths, or temporarily relax the env knob (S2_RISK_DIM_THRESHOLD=1.01) and restart sidecar. Don't disable globally — you'll lose the signal that pointed at a real issue.

Testing

pytest -q -m "not live"                          # offline suite
RC_LIVE=1 pytest -q tests/test_scaleway_smoke.py # live Scaleway round-trip (optional)
bash scripts/test-prototype.sh                   # full e2e gate

Contributing

Yes please.

1. Spec first. Update docs/PLAN.md before writing code; declare acceptance criteria explicitly.
2. Add a kanban entry. Edit board/board.json or open an issue.
3. Self-verify. Before opening a PR:
   • python3 -m py_compile $(git ls-files '*.py') exits 0
   • python3 -m pytest -m "not live" is green
   • bash -n scripts/*.sh clean
   • python3 -c "import json; json.load(open('.claude/settings.json'))" passes
4. Don't break the contracts. HTTP /score, ImpactReport JSON, MCP tool signature are public.
5. Use real things. Real Mamba weights, real Tree-sitter grammars. Mocks only in tests.

Roadmap

The current shipped surface goes well past verification-only — the gate participates in planning, mines its own calibration corpus, and runs concurrent calibration in shadow mode. The next phase is closing the loop: iterative repair so Claude re-proposes against the repair hint until pass-or-yield.

Read thoughts/shared/research/ for the risk-vector-delta-refactor + coherence-delta-calibration write-ups and docs/PLAN.md for the spec.

Shipped (see git log --grep='^feat')

• P-1 — Day-zero ergonomics: magic-comment escapes, RC_BYPASS_NEXT kill switch, rc CLI.
• P0 — Validation harness: embedder fitness test, calibration corpus, golden set, shadow-mode wiring.
• P1 — Plan-time SSM scoring + plan→code coherence gate, mock-detector heuristics.
• P2 — Generative repair head: Qwen2.5-Coder-1.5B via MLX / Scaleway, behind RC_REASONER_BACKEND.
• P3 — Calibration: Mahalanobis over 8-dim risk space, per-kind shrinkage, Page-Hinkley monthly recalibration.
• P4 — Calibration corpus + golden set + OOD detector + shadow-mode hardening.
• P5 — Sidecar broker + supervisor + grounding eval on 200 labeled pairs.
• P5 round-3 hardening — 4-reviewer findings closed: per-dim Pareto epsilon, semantic safety net, stderr truncation contract, rules.yaml fail-closed (b7f0517).
• P5 grounding pairs v2 — judge-relabeled high-confidence subset; live κ=0.74 on v2 (kin-judge contaminated, gate advisory) (4ed3245).
• P5 grounding pairs v3 — cross-family κ dataset — 131 high-confidence pairs (200 input) × 3 judges (devstral + llama-3.3-70b + mistral-small); max pairwise-κ = 0.6998 < 0.7 independence gate; sentinel qwen_kappa_gate.json reports gate_pass=true at κ=0.8025. Replaces the kin-judge-contaminated v2.
• Linux systemd user-unit recipe — copy-paste ~/.config/systemd/user/reasoning-core-sidecar.service documented inline in §6 (mirrors the macOS launchd supervisor).
• P7 — Calibration concurrent with shadow mode (Mahalanobis + Ledoit-Wolf shrinkage + empirical-Bayes per-kind anchor).
• P7 supervisor watcher — consumes recalibrate.signal for hot-refit without restart (5313498).
• Iter-2 readiness blockers closed — GUARDED_PATHS extended to all hook helpers + supervisor + gen_client + calibration; binomial sign-test in eval/stats.py for the falsifiable goal (6a921ce).
• CI eval workflow stabilized — sharded-safetensors fallback, lazy prefetch, --run-id arg, contents:write permission (c452cb4/c0118a4/4997ec7/dcd3598/1738b57/1bc2718).

Open

• Synthesize-Check-Refine loop — Phase 3 of 2026-05-06-system-2-loop-closure.md: on block, auto-call the generative critic, re-score with sidecar, emit validated repair as stderr hint. Iteration is server-side; agent never sees the loop.
• Hybrid symbolic gating (ADR injection) — Phase 2 of the same plan: .reasoning-core/rules.yaml + _rule_engine.py for layered-import / forbidden-pattern / metric-threshold rules co-emitted with the neural risk vector through the same exit-2 pipe.
• TTFV (<15 min) + drift visualization — Phase 1: rc audit-history (last N commits, what would have been blocked) + rc viz Mermaid sparkline + npx reasoning-core init one-line installer.
• CUDA / MLX kernels for the Mamba forward pass (currently CPU-only; p95 ~5s).
• SSE /score/stream + Prometheus textfmt /metrics (current /metrics returns JSON; SSE endpoint not yet wired).
• Pre-commit variant so non-Claude editors are also gated.
• Mamba-3 watch + Plan-B Mamba-2-2.7B fallback — Phase 4: HOLD on Mamba-3 (no public HF checkpoint as of 2026-05; CUDA-only kernels; only 1/8 risk dims uses SSM embedding). Plan-B fallback ships behind RC_USE_MAMBA2_2_7B=1.

Roadmap source of truth: docs/PLAN.md + thoughts/shared/research/.

Acknowledgements + License

• Mamba — Albert Gu & Tri Dao.
• Tree-sitter — Max Brunsfeld.
• Model Context Protocol — Anthropic.
• FastMCP — Jeremiah Lowin.
• direnv — repo-scoped env without leaking into other shells.

MIT © Jakub Sikora — use it, fork it, ship something better.