One of the strangest workflows in modern software development looks like this:
- Ask ChatGPT a question.
- Realize it doesn't have enough context.
- Copy a few files.
- Add architecture notes.
- Paste recent changes.
- Explain what you're trying to do.
- Hope you included the right information.
The problem isn't the AI.
The problem is context retrieval.
After building TokenCap's Knowledge Graph, Debug Handoff, Change Intelligence, and Context Packing systems, I realized something:
We had become very good at generating context.
We were still terrible at finding it.
That's what led to TokenCap v0.7 — Smart Retrieval Engine.
The Question
Instead of asking:
Which files should I send to the AI?
What if you could simply ask:
tokencap ask "How does authentication work?"
and let the tool figure out the rest?
How It Works
The new retrieval engine searches across multiple sources of project intelligence:
- Knowledge Graph relationships
- Dependency chains
- Project Memory
- Recent code changes
- Risk analysis
- Impact scores
It doesn't just find matching files.
It finds the files that matter.
Beyond Keyword Search
A search for:
tokencap ask "debug login redirect"
automatically switches into debug mode.
A search for:
tokencap ask "review payment flow"
switches into review mode.
A search for:
tokencap ask "explain dashboard architecture"
switches into architecture mode.
The retrieval strategy changes based on what you're trying to achieve.
Following the Connections
One thing I wanted to avoid was returning isolated files.
Real systems are connected.
If a file is relevant, its dependencies are often relevant too.
TokenCap traverses the graph to find related files, dependents, and neighboring modules before building the final context package.
Context Isn't Free
Even after finding the right files, another challenge remains:
Token budgets.
The retrieval engine reuses TokenCap's Context Packing system to decide what deserves:
- Full source code
- Structural outlines
- Summaries
- References only
This allows larger repositories to fit into practical AI context limits without losing important information.
From Context Generation to Context Retrieval
The evolution of TokenCap has been interesting:
- v0.1 — Project Snapshot
- v0.2 — Knowledge Graph & Context Memory
- v0.3 — Debug Handoff
- v0.4 — Change Intelligence
- v0.5 — Context Packing
- v0.6 — Graph Intelligence
- v0.7 — Smart Retrieval
For the first six versions, the focus was:
Generate better context.
With v0.7, the focus becomes:
Retrieve the right context.
That's a subtle difference.
But I think it's the difference between AI tools that know code and AI tools that understand projects.
Try It
Website: tokencap.vansharora.app
NPM: https://www.npmjs.com/package/tokencap
npm install -g tokencap
Then ask your repository a question:
tokencap ask "How does authentication work?"
Instead of manually assembling context, let the repository tell the AI what matters.
Feedback, ideas, and contributions are always welcome.