Originally published on andrew.ooo — visit the original for any updates, code snippets that aged out, or follow-up posts.

TL;DR

Forge is a Python reliability layer for self-hosted LLM tool-calling, built by Antoine Zambelli, AI Director at Texas Instruments. It hit the Hacker News front page on May 19, 2026 with 206 points and a tagline that did the heavy lifting on its own: "Guardrails take an 8B model from 53% to 99% on agentic tasks."

The headline result, drawn from Forge's own 26-scenario eval suite:

• Bare 8B local model: ~53% on multi-step agentic workflows
• Same 8B model + Forge guardrails: 99.3%
• Claude Sonnet, no guardrails: 87.2%
• Claude Sonnet + Forge: ~100%

That second-to-last line is the one that makes people sit up: a $0 local 8B model on a $600 GPU, wrapped in Forge, beats a frontier API used the normal way. The gap between bare-metal local inference and a cloud API closes to less than 1 percentage point when both sides run through Forge. The framework, the ablation study, and the methodology are written up in an IEEE-track paper (Zambelli 2026, DOI: 10.1145/3786335.3813193).

Key facts:

• MIT licensed, pip-installable as forge-guardrails
• Backends: Ollama, llama-server (llama.cpp), Llamafile, Anthropic
• Top eval config: Ministral-3 8B Instruct Q8 on llama-server
• Three usage modes: full WorkflowRunner, middleware-only, drop-in OpenAI-compatible proxy
• Core trick: rescue parsing, retry nudges, step enforcement, tiered context compaction, plus a synthetic respond tool that keeps small models in tool-calling mode
• 865 deterministic unit tests + 26-scenario eval harness
• Status: active (Python 3.12+), authored solo, real paper backing it

If you have a 24 GB GPU sitting in a closet and you've been waiting for the moment when "self-hosted agents" stops meaning "self-hosted toys," this is the moment.

Quick Reference

Why this matters

For most of 2025 and early 2026, the conventional wisdom on local agents was: nice for chat, useless for tool calling. Anyone who plugged a local 8B into a five-step workflow watched it confidently call get_weatherr with a typo, return text instead of a JSON tool call, forget which step it was on, and time out somewhere in the middle. Real agentic use stayed on frontier APIs.

Forge's contribution is to show that the gap is not really a model problem — it's a harness problem. The model knows how to call tools. It just needs:

1. Permission to fail and try again (retry nudges)
2. A parser that can salvage a malformed call instead of throwing
3. A reminder of which step it's on when it wanders
4. A context manager that doesn't quietly truncate the last tool result
5. A way to emit text without leaving tool-calling mode

Stack those five things and the same weights that score 53% raw score 99%. That's the entire argument, and Forge implements every piece of it as composable middleware so you can use the parts you need.

Install in three minutes

# core only
pip install forge-guardrails

# plus the Anthropic client
pip install "forge-guardrails[anthropic]"

You also need a backend. Forge is most opinionated about llama-server (its top-10 eval configs all run on it), but Ollama is the easiest on-ramp:

# Ollama path — easiest
ollama pull ministral-3:8b-instruct-2512-q4_K_M

# llama-server path — best performance
llama-server -m path/to/Ministral-3-8B-Instruct-2512-Q8_0.gguf \
  --jinja -ngl 999 --port 8080

The --jinja flag matters. It's what enables native function calling in llama.cpp, and without it you fall back to prompt-injected tool calling, which is measurably worse on the harder eval tier.

A real workflow in 30 lines

This is the canonical Forge example, lightly trimmed from the README:

import asyncio
from pydantic import BaseModel, Field
from forge import (
    Workflow, ToolDef, ToolSpec,
    WorkflowRunner, OllamaClient,
    ContextManager, TieredCompact,
)

def get_weather(city: str) -> str:
    return f"72°F and sunny in {city}"

class GetWeatherParams(BaseModel):
    city: str = Field(description="City name")

workflow = Workflow(
    name="weather",
    description="Look up weather for a city.",
    tools={
        "get_weather": ToolDef(
            spec=ToolSpec(
                name="get_weather",
                description="Get current weather",
                parameters=GetWeatherParams,
            ),
            callable=get_weather,
        ),
    },
    required_steps=[],
    terminal_tool="get_weather",
    system_prompt_template="You are a helpful assistant. Use the available tools to answer the user.",
)

async def main():
    client = OllamaClient(
        model="ministral-3:8b-instruct-2512-q4_K_M",
        recommended_sampling=True,
    )
    ctx = ContextManager(
        strategy=TieredCompact(keep_recent=2),
        budget_tokens=8192,
    )
    runner = WorkflowRunner(client=client, context_manager=ctx)
    await runner.run(workflow, "What's the weather in Paris?")

asyncio.run(main())

What's quietly happening behind that runner.run call:

• ResponseValidator checks every model response against the tool schema, flags malformed JSON, and routes failures to the nudge system instead of bubbling them up.
• StepEnforcer tracks required_steps (none in this example, but typical workflows have them) and refuses to terminate until they're satisfied.
• TieredCompact keeps the last 2 turns verbatim and compacts older turns when the context budget gets tight, so a runaway loop doesn't quietly drop the original user request.
• The retry nudge templates in forge/prompts/nudges.py produce small, surgical "you produced invalid output, here's why, try again" messages instead of restarting the loop.
• recommended_sampling=True auto-applies the sampling params the model card recommends, which Zambelli found made a measurable difference in tool-call accuracy on small models.

This pattern — defining a Workflow, picking a backend client, attaching a ContextManager, and handing both to WorkflowRunner — is the high-level entry point. If you'd rather keep your own orchestration loop, Forge also ships a middleware mode where you call its guardrails directly on responses you've already gotten back from a model.

The drop-in proxy mode (this is the killer feature)

This is the bit that will pull people in even if they have no interest in rewriting their stack:

# external mode — you run llama-server, Forge wraps it
python -m forge.proxy --backend-url http://localhost:8080 --port 8081

# managed mode — Forge starts llama-server for you
python -m forge.proxy --backend llamaserver \
  --gguf path/to/Ministral-3-8B-Instruct-2512-Q8_0.gguf --port 8081

Now point any OpenAI-compatible client — opencode, Continue, aider, your homegrown agent loop — at http://localhost:8081/v1 and you get the full Forge guardrail stack for free. The client thinks it's talking to a smarter model.

Internally the proxy does one slightly diabolical thing: when a request contains tools, it injects a synthetic respond tool. The model is then expected to call respond(message="...") instead of emitting bare text. This keeps it in tool-calling mode at all times — which is where Forge's guardrails apply — and the synthetic call is stripped from the outbound response so the client never sees it. ADR-013 in the repo has the full reasoning, but the short version: 8B models cannot be trusted to decide between "emit text" and "emit a tool call." Force them to always pick a tool, and accuracy jumps.

The 26-scenario eval suite

Forge ships its own benchmark. Two tiers:

• OG-18: 18 baseline scenarios covering single tool calls, multi-step workflows, parameter validation, and recovery from injected errors.
• advanced_reasoning: 8 harder scenarios — chained dependencies, branching logic, ambiguous user prompts, partial information.

Run it yourself:

python -m tests.eval.eval_runner \
  --backend llamafile \
  --llamafile-mode prompt \
  --gguf "path/to/Ministral-3-8B-Instruct-2512-Q8_0.gguf" \
  --runs 10 --stream --verbose

# batch eval across configs
python -m tests.eval.batch_eval --config all --runs 50

# HTML dashboard
python -m tests.eval.report eval_results.jsonl

The published numbers:

The "99.3% on agentic tasks" headline number comes from a narrower slice of the benchmark — strict tool-calling reliability, not the full combined score. Read the paper for the precise breakdown; Zambelli is straightforward about which number means what.

One of his most interesting eval findings, surfaced in the HN thread: the serving backend matters as much as the model. Mistral-Nemo 12B weights produce 7% accuracy on llama-server with native function calling and 83% on Llamafile in prompt-injected mode. Same weights. The difference is entirely in how the chat template hands the tool schema to the model.

Community reaction

The HN thread had real engineers in it. A few notes worth pulling out:

On the "small models with harnesses beat big models" thesis: "I've been saying for a while that given a proper harness, small local models can perform incredibly well. When you have a system that can try everything, it will eventually get it right as long as you can prevent it from getting it wrong in the meantime." (HN user, 48200359)

Zambelli's response: "Essentially, yes that's right! There's some subtlety in how to let it know it was wrong (returning things as tool errors because it trained on that), but that's the gist of it — sort of a self-correcting architecture."

The most useful piece of pushback, from a heavy local-LLM user: "Specifically for agentic workflows and local models, accuracy around function/tool calling hasn't been a problem for me now for about 6 - 12 months, personally, since around QwenCoder3. The main issue is context management and the impact on timing... It looks like your work adds layers and wrappers like guard rails and retries. This would make my local model experience - specifically for agents - unusable because of the delays it would add."

This is the right critique. Forge improves accuracy but adds round-trips. Zambelli responded by floating a new metric he's going to start tracking: ETTWS — Estimated Time To Working Solution. A retry loop that succeeds on attempt three with three small models calls might still beat a single shot that gets 53% accuracy and then forces you to restart the whole workflow. But you need the data to defend that claim, and right now it's not in the paper.

On the framing: "This is a thousand unusually smart monkeys who speak every major human language fluently and are proficient in every major programming language, but sometimes still make bizarre mistakes and need to be put back on track." Forge is, in that framing, the trainer with the clicker.

Honest limitations

This is not a finished product. Things to know before you bet a workflow on it:

1. No public latency benchmarks. All the published numbers are accuracy. The HN thread surfaced this and Zambelli acknowledged it. If you're running an interactive agent, you need to time-box your retries and prove to yourself that the wall-clock-to-success is better than your current setup.

2. Python only. No JS, Go, or Rust bindings. If your stack is Node-first, you're either standing up a sidecar or waiting.

3. Model coverage is uneven. The eval suite is heavily skewed toward the Ministral-3 family and Mistral-Nemo. Qwen, Llama 3.x, and Gemma configurations exist but are less battle-tested. The Model Guide in the repo is honest about which combos have been graded.

4. The synthetic respond tool is a workaround, not a fix. Some users will be uncomfortable that the proxy injects an extra tool into every request. ADR-013 is worth reading before you adopt this in production.

5. Multi-agent and slot-sharing support is brand new. SlotWorker (priority-queued access to a shared inference slot) is in the repo but isn't yet a documented production pattern.

6. Active solo project. It's a one-maintainer project (Zambelli, plus an academic paper). If you adopt it, plan to read the code.

Forge vs. the alternatives

The cleanest mental model: Outlines/Instructor enforce schema, vLLM constrained decoding enforces tokens, LangGraph orchestrates flows, Forge enforces a workflow. They're complementary, not competitive.

Who should adopt this today

• Solo developers running local agents on a 24–48 GB GPU. You'll feel the reliability jump on day one.
• Teams building privacy-sensitive agentic products (legal, medical, on-prem) where a frontier API isn't on the table and the current local stack is too flaky.
• Research groups doing ablation studies on tool-calling reliability — Forge's eval harness is already publication-quality.
• Anyone running opencode, Continue, or aider against a local model and wishing it were less stupid. The proxy mode is a 30-second adoption.

Hold off if:

• You need sub-second latency on every step and can't tolerate retry overhead.
• Your stack is Node-only and you can't stomach a Python sidecar.
• You're already paying for frontier API access and your reliability problem is "Claude refused to call my tool once last week" — Forge is overkill for that.

FAQ

Q: Does Forge work with Qwen, Llama 3, or Gemma — or only Ministral?
A: It works with anything its backends support (Ollama, llama-server, Llamafile, Anthropic). The published eval numbers happen to be best on Ministral-3 8B Q8, but the framework is model-agnostic. The Model Guide in the repo is honest about which model + backend combos have been graded and which are "should work, untested."

Q: How much slower is a Forge-wrapped run vs. a raw model call?
A: There are no public latency numbers yet, which is the main critique on HN. In practice, you're trading wall-clock for completion rate: a single-shot 8B might return in 800ms and be wrong; a Forge-wrapped run might take 2–4 seconds and finish the workflow correctly. Zambelli is adding an ETTWS (Estimated Time To Working Solution) metric to the next paper rev specifically to address this.

Q: Do I have to use the WorkflowRunner, or can I just bolt the guardrails onto my existing loop?
A: You can absolutely just use the middleware. examples/foreign_loop.py in the repo shows how to call ResponseValidator, the nudge system, and the context manager directly. The full WorkflowRunner is convenient but optional.

Q: What's the respond tool actually for?
A: 8B models trained on tool-calling datasets get confused when asked to choose between emitting plain text and emitting a tool call. Many will emit text when they should call a tool, or call a tool when they should just answer. The synthetic respond tool removes the choice: there's always a tool to call, even for plain answers. The proxy strips it from the response on the way out so clients see normal text. ADR-013 in the repo has the full analysis and the eval data behind the decision.

Q: Is the IEEE paper open-access or paywalled?
A: The DOI (10.1145/3786335.3813193) may take a moment to resolve depending on publisher timing, but there's a pre-publication preprint kept in the repo at docs/forge_ieee_preprint.pdf. Cite the DOI version, read the preprint.

Q: Does it support streaming?
A: Yes — StreamChunk and SSE streaming are first-class in the client and proxy layers. The proxy server can stream chunks to OpenAI-compatible clients while still applying guardrails on completion boundaries.

Q: Can I use it with Anthropic as a baseline / hybrid?
A: Yes. pip install "forge-guardrails[anthropic]" and use AnthropicClient as the backend. Useful for A/B tests against your local config or for hybrid workflows where part of the chain runs on Claude and part runs locally.

Bottom line

Forge is the most concrete answer yet to the question "is self-hosted agentic AI actually viable, or is it always going to be the toy alternative?" On the published evidence — and the IEEE paper behind it — the answer is: with the right harness, an 8B local model in a $600 GPU is functionally on par with a frontier API. That's a tectonic shift for anyone who has been waiting for the moment when running real agents at home stops feeling masochistic.

The latency story is unfinished. The model coverage is uneven. It's a solo project. None of that changes the headline result, which is one of the cleanest demonstrations of "framework matters more than weights" published this year.

If you have a GPU, install it tonight. Point your existing agent client at the proxy and see what happens. The setup cost is fifteen minutes; the upside is the local agent stack you've been waiting on.

Repo: github.com/antoinezambelli/forge
Paper: Zambelli 2026, DOI 10.1145/3786335.3813193
HN thread: news.ycombinator.com/item?id=48192383