The Token Tax Problem: How I Built a Super Memory Layer for AI Coding Assistants

We Solved the Wrong Problem First

When AI coding assistants arrived, we celebrated. Faster delivery. Less repetitive work. Developers doing more meaningful things.

Then the invoices arrived.

Token utilization had quietly become one of the fastest-growing line items in engineering costs. Every session, every agent, every code suggestion — all of it burning through context tokens. And the root cause was embarrassingly simple: we were paying for AI tools to re-learn our codebase from scratch, over and over again.

Round One: The Obvious Fixes

We started with the basics. Things that genuinely helped:

• Context window hygiene — Being deliberate about what goes into context rather than dumping entire file trees at every agent invocation
• Model switching — Using faster, cheaper models for repetitive low-complexity tasks and reserving powerful models for architecture decisions and complex debugging
• Preprocessed context — Writing structured markdown instruction files that encode team conventions once and reuse them everywhere, instead of expecting agents to infer them from raw code
• Scoped agents — Purpose-built agents for specific tasks (test generation, code review, planning) rather than one general-purpose agent doing everything

These helped. But they didn't solve the fundamental issue. Agents were still spending tokens exploring the codebase before doing any real work.

We needed something closer to a cache layer.

The Core Idea: A Super Memory Layer

The inspiration came from Andrej Karpathy's concept of the LLM Wiki — the idea that an AI system benefits enormously from a persistent, structured knowledge index rather than re-reading raw source on every request.

Think of it like CloudFront or Redis in front of your origin server.

Instead of every agent making expensive round trips into raw source code, they read from a pre-built knowledge graph. That graph becomes a shared memory layer — a single source of architectural truth accessible by any AI tool: Copilot, Factory, Claude, Cursor, or whatever comes next.

For the implementation, I used Graphify (github.com/safishamsi/graphify), an open-source tool that converts a codebase into a knowledge graph:

• Nodes — functions, components, hooks, utilities
• Edges — relationships between them (imports, calls, dependencies)
• Output — a plain-language report, interactive visualization, and GraphRAG-ready JSON

The POC: Steps We Actually Followed

Step 1 — Full Codebase Attempt (Hit a Wall)

First instinct: run it on the entire codebase at once.

The corpus exceeded the tool's recommended limits immediately (~900+ files). This is actually a healthy constraint — feeding an LLM a massive undifferentiated codebase produces poor graph quality anyway.

Lesson: Large codebases need a per-module strategy.

Step 2 — Module-by-Module Analysis

We split the codebase by independent modules and ran the graph pipeline on each one separately.

Each run was completely free — Graphify's AST extraction is pure static analysis with zero LLM API calls. The graph structure emerged from the code itself:

Step 3 — Debugging the Tool Itself

During a couple of runs, report generation failed due to API signature changes between Graphify versions. We patched the calls and kept moving.

Lesson: Pin your open-source tooling versions. APIs shift.

Step 4 — Merging Module Graphs

With individual module graphs ready, we wrote a merge script to combine them into a single unified knowledge graph.

First attempt had a subtle bug — the script accidentally read the same module's extract file multiple times (once per module), producing a graph full of duplicates. We caught it because all sets of nodes were identical.

Fix: Rebuilt the merge from each module's actual AST cache files, prefixing node IDs with the module name to prevent collisions:

node_id = f'{module_name}::{original_node_id}'

Correct merged result:

• 1,600+ unique nodes across all modules
• 4,000+ edges
• 28 structural communities detected
• Zero LLM tokens consumed to build it

Step 5 — Discovering the God Nodes

The most valuable output wasn't the graph itself — it was what the graph revealed.

God nodes are the most connected abstractions in the codebase. The functions, utilities, and components that everything else depends on. Most experienced developers know these intuitively but have never seen them mapped explicitly.

Once you know your god nodes, you can:

• Prioritise documentation specifically for these high-impact functions
• Instruct agents to proceed carefully whenever changes touch them
• Use them as architectural anchors in any context window

Step 6 — Wiring the Graph into Agent Instructions

We updated the agent instruction files used by each tool (GitHub Copilot, Factory/droid, etc.) to point at the merged graph report as their primary architecture reference:

Before answering architecture or codebase questions,
read the merged graph report at graphify-out/GRAPH_REPORT_MERGED.md

This means any agent that loads these instructions starts with architectural knowledge already loaded — without scanning source files to build that understanding themselves.

A 9KB markdown report replacing several megabytes of source scanning. Every session.

Step 7 — Running the Token Experiment

To quantify the impact, we set up an A/B test. We commented out the graph instructions from all agent configuration files, then ran identical tasks in both configurations and compared token consumption.

<!-- graphify section disabled for token utilization analysis
     re-enable when experiment is complete -->

Results from the experiment will follow in a separate post.

The Three Outputs: What Agents Actually Consume

Every Graphify run produces three files, each serving a different consumer:

For token efficiency, agents read only the markdown (~9KB). The JSON is available for tools that can query it selectively.

Honest Pros and Cons

✅ What Works

• Zero build cost — AST extraction consumes no LLM tokens
• Tool-agnostic — Works with any tool that reads files (Copilot, Factory, Claude, Cursor)
• Shared memory — One knowledge base, many consumers; no duplication of analysis
• God node awareness — Agents automatically know which abstractions are highest-impact
• Community detection — Related code clusters surface naturally without manual documentation

⚠️ What Doesn't

• Stale risk — Graph must be regenerated after structural changes; a stale graph actively misleads agents
• Velocity tension — Codebases with rapid daily structural changes will find frequent regeneration expensive in time, even if not in tokens
• Corpus size limits — Large repos must be split by module; cross-module edges are inferred, not extracted
• No semantic understanding — AST-only extraction misses business intent and domain meaning; semantic extraction adds LLM cost
• Merge complexity — Combining module graphs requires care; duplicate nodes and ID collisions are easy mistakes to make

Suggestions to Take This Further

1. Incremental updates — Re-extract only changed files after each commit, not the full module
2. Automate in CI — Regenerate affected module graphs as a post-merge pipeline step, triggered only when source files change
3. Selective semantic enrichment — Run LLM-assisted extraction only on shared utilities and god nodes, not on every file
4. Add a wiki navigation layer — Generate a navigable index.md so agents load only the relevant section of the graph rather than the full report
5. Commit only the report — The markdown report is the token-saver; the JSON and HTML can stay gitignored to avoid bloating the repo

The Bigger Picture

Token cost is the new technical debt of the AI-assisted development era.

Every pattern that reduces it — pre-processed context, structured instructions, scoped agents — points in the same direction:

Give agents knowledge, not raw data.

A project knowledge graph is one concrete implementation of that principle. It is not magic, and it is not free to maintain. But as a cache layer between your codebase and your AI tools, it fundamentally changes the economics of agent-assisted development.

The experiment is ongoing. I'll share the token comparison numbers once the A/B test wraps up.

If you're working on similar token efficiency problems or have taken a different approach, I'd love to hear about it in the comments.

Resources

• 🔧 Graphify on GitHub
• 💡 Concept inspiration: Andrej Karpathy — LLM Wiki
• 📊 Token experiment results: coming soon