LLM Foundry: the boring stack that makes an LLM actually useful

Most AI projects are built backwards.

People start with the model and only later discover they needed a memory system, semantic retrieval, tool use, tests, and a fallback plan for when one provider decides to nap for no visible reason.

That is the part I care about now.

LLM Foundry is the workshop around an LLM — not the model itself. It is the layer that makes a model useful for actual work instead of just looking smart in a demo.

What changed

The current version now has a few things worth showing instead of just claiming:

• semantic retrieval backed by embeddings, so memory search is not just keyword matching
• multi-provider support for OpenAI-compatible endpoints, Anthropic, Hugging Face, and failover bundles
• compression + memory so long tasks can be shrunk into a compact working context
• agent traces that can be exported into training data
• benchmark + harness runs so the system is testable instead of vibes-based

That last bit matters more than people like to admit.

If a system cannot be tested, it is not “advanced”. It is just expensive.

The core idea

A useful model stack is not one prompt and a prayer.

It is usually:

1. read the task
2. recover relevant memory
3. compress the clutter
4. ask the model
5. check the answer
6. use tools if needed
7. save traces
8. benchmark the result

That is the difference between a chatbot and something you might actually trust on real work.

The honest part: orchestration helps, but it does not create capability from thin air

This part matters, because the AI world does itself a lot of damage by overpromising.

If a base model is bad at reasoning, orchestration will not magically make it frontier-grade. You can improve its behaviour, reliability, recall, and workflow quality. You cannot conjure missing intelligence out of nowhere.

That is not a flaw in the system. That is just reality.

What orchestration can do is make a decent model much more useful:

• it sees less irrelevant text
• it retrieves the right context more often
• it can call tools instead of guessing
• it can be checked and scored
• its traces can become training data later

That is the real win.

Proof, not poetry

Here is the validation package I used while testing the repo:

• Live report: https://zo.pub/man42/llm-foundry
• Screenshot 1: https://zo.pub/man42/llm-foundry/top.png
• Screenshot 2: https://zo.pub/man42/llm-foundry/mid.png
• Screenshot 3: https://zo.pub/man42/llm-foundry/bottom.png

The numbers

That benchmark pass rate is not a brag. It is a baseline. The point is that the system is measurable, and therefore improvable.

Screenshots

Why semantic retrieval matters here

I wanted the memory system to work for normal tasks, not just demos.

So the retrieval layer is now embedding-based. That means the system can look for relevant context semantically, not just by literal word match.

That matters when the task wording changes but the meaning does not.

In plain English: it is much harder for the assistant to miss the useful note just because you phrased the request differently.

That is a small change with outsized effect.

What I’m actually trying to build

The goal is not “a model wrapper”. The goal is a practical operating layer for LLM work:

• a model can be local or remote
• the backend can be OpenAI-compatible or Anthropic
• memory can be compacted and reused
• traces can become training data
• benchmarks can tell you whether anything improved

That is the kind of infrastructure that makes a model usable for long jobs, research, and product workflows.

Code and proof

• GitHub repo: https://github.com/AmSach/llm-foundry
• GitHub profile: https://github.com/AmSach
• Proof pack: https://zo.pub/man42/llm-foundry

Find me here too

• Instagram: https://www.instagram.com/i.amsach
• LinkedIn: https://www.linkedin.com/in/theamansachan