Note: This post contains architecture diagrams. For the fully rendered version, visit the original post.
In the previous post, I laid out the full six-layer architecture of an open-source AI coding IDE. Today we start building it.
This post ships the first real component of Dhi (धी) — an open-source AI coding IDE built entirely on open-source models. Dhi means pure intellect in Sanskrit, from the Gayatri Mantra. The name fits: the goal is an IDE that gives you genuine intelligence over your codebase — with no API keys, no token pricing, and no closed-source inference backend.
By the end of this post you will have:
- A working FIM autocomplete engine running on your laptop
- A Tree-sitter-based semantic chunker for Python and TypeScript
- A Chroma vector store with nomic-embed-text-v1.5 embeddings
- A StarCoder2-3B inference server via Ollama
- A VS Code extension with ghost-text inline completions
Everything runs with docker compose up. The full code is at github.com/sochaty/dhi — tag post-1.
What FIM Actually Is
Most developers think of autocomplete as next-token prediction: the model sees everything before the cursor and predicts what comes next. That is how GPT-2 works. It is not how a modern AI coding assistant works.
Real autocomplete in 2026 is fill-in-the-middle (FIM): the model sees both the prefix (everything before the cursor) and the suffix (everything after), then generates the completion that bridges them. This is far more accurate because the model knows what the code is supposed to arrive at, not just what it started from.
The three special tokens that make FIM work:
| Token | Contains | Direction |
|---|---|---|
<fim_prefix> |
Everything before the cursor | → feeds into model |
<fim_suffix> |
Everything after the cursor | → feeds into model |
<fim_middle> |
The completion | ← model generates this |
StarCoder2, DeepSeek-Coder, and Qwen2.5-Coder all support FIM natively. This is non-negotiable for a production autocomplete engine — a model without native FIM support gives noticeably worse inline completions.
The full request flow for Dhi's autocomplete engine:
VS Code Editor (cursor position event)
↓ 150ms debounce
Context Assembler
↓ ↓
fim_prefix fim_suffix
retrieved chunks (top-3) file below cursor
file above cursor
↓ ↓
Ollama (StarCoder2-3B)
↓
Ghost text in editor
The critical design decision here: the prefix is not just the current file. It includes retrieved chunks from the rest of the repository. Without that context, the model cannot complete a function call using a helper defined in another file.
Bootstrapping Dhi
Clone the repo and start the CPU stack:
git clone https://github.com/sochaty/dhi
cd dhi
git checkout post-1
docker compose up
The docker-compose.yml at this tag runs three services:
services:
server:
build: ./server
ports: ["8000:8000"]
environment:
- CHROMA_HOST=chroma
- OLLAMA_HOST=ollama
depends_on: [chroma, ollama]
chroma:
image: chromadb/chroma:0.5.0
volumes: ["chroma_data:/chroma/chroma"]
ollama:
image: ollama/ollama:latest
volumes: ["ollama_models:/root/.ollama"]
entrypoint: ["/bin/sh", "-c", "ollama serve & sleep 5 && ollama pull starcoder2:3b && wait"]
On first start, Ollama pulls StarCoder2-3B (~1.7GB). Subsequent starts are instant. The full repo directory structure at post-1:
dhi/
├── docker-compose.yml
├── extension/
│ └── src/completion/provider.ts
└── server/
├── main.py
├── inference/fim.py
└── rag/
├── chunker.py
└── store.py
Layer 1: Tree-sitter Semantic Chunking
The naive approach to chunking code for a vector store is splitting on character count — every 500 characters becomes a chunk. This is wrong for two reasons: it cuts function bodies in the middle (destroying semantic meaning) and it groups unrelated code together (polluting retrieval).
Dhi uses Tree-sitter to split on semantic boundaries. A function definition becomes one chunk. An import block becomes one chunk. A class body becomes one chunk. Each chunk is self-contained and contextually coherent.
Source File (.py / .ts)
↓
Tree-sitter Parser → Syntax Tree
↓
import_block → 1 chunk
function_definition → 1 chunk
class_definition → 1 chunk
↓
+ metadata (file path · line range · language)
↓
nomic-embed-text-v1.5 (768-dim)
↓
Chroma
Here is server/rag/chunker.py — the full implementation for Python and TypeScript:
from dataclasses import dataclass
from pathlib import Path
from typing import Generator
import tree_sitter_python as tspython
import tree_sitter_typescript as tstypescript
from tree_sitter import Language, Parser
PY_LANGUAGE = Language(tspython.language())
TS_LANGUAGE = Language(tstypescript.language_typescript())
# Node types that become independent chunks
CHUNK_NODE_TYPES = {
"python": {
"import_statement", "import_from_statement",
"function_definition", "async_function_definition",
"class_definition",
},
"typescript": {
"import_declaration", "import_statement",
"function_declaration", "arrow_function",
"class_declaration", "method_definition",
"interface_declaration", "type_alias_declaration",
},
}
@dataclass
class Chunk:
text: str
file_path: str
start_line: int
end_line: int
language: str
node_type: str
def _get_parser(language: str) -> Parser:
parser = Parser()
if language == "python":
parser.set_language(PY_LANGUAGE)
else:
parser.set_language(TS_LANGUAGE)
return parser
def _detect_language(path: str) -> str | None:
suffix = Path(path).suffix
if suffix == ".py":
return "python"
if suffix in (".ts", ".tsx"):
return "typescript"
return None
def chunk_file(file_path: str) -> Generator[Chunk, None, None]:
language = _detect_language(file_path)
if language is None:
return
source = Path(file_path).read_bytes()
parser = _get_parser(language)
tree = parser.parse(source)
target_types = CHUNK_NODE_TYPES[language]
def walk(node):
if node.type in target_types:
text = source[node.start_byte:node.end_byte].decode("utf-8", errors="replace")
yield Chunk(
text=text,
file_path=file_path,
start_line=node.start_point[0] + 1,
end_line=node.end_point[0] + 1,
language=language,
node_type=node.type,
)
else:
for child in node.children:
yield from walk(child)
yield from walk(tree.root_node)
Two things worth noting:
Greedy top-level chunking. The walk function yields a node and stops recursing into it. A class body becomes one chunk — it does not also yield its individual methods as separate chunks. This keeps retrieval units large enough to be coherent.
Language-specific grammar packages. tree_sitter_python and tree_sitter_typescript are PyPI packages that ship pre-compiled grammars. No C compilation step required, which matters for Docker images.
Layer 2: Embedding and Vector Storage
Each chunk is embedded with nomic-embed-text-v1.5 — a 768-dimension model that runs locally via Ollama. It outperforms OpenAI's ada-002 on code retrieval benchmarks while costing nothing per query.
Here is server/rag/store.py:
import hashlib
import os
from typing import Sequence
import chromadb
import httpx
CHROMA_HOST = os.getenv("CHROMA_HOST", "localhost")
OLLAMA_HOST = os.getenv("OLLAMA_HOST", "localhost")
COLLECTION_NAME = "dhi_chunks"
EMBED_MODEL = "nomic-embed-text"
def _embed(texts: list[str]) -> list[list[float]]:
resp = httpx.post(
f"http://{OLLAMA_HOST}:11434/api/embed",
json={"model": EMBED_MODEL, "input": texts},
timeout=30,
)
resp.raise_for_status()
return resp.json()["embeddings"]
def _chunk_id(chunk) -> str:
key = f"{chunk.file_path}:{chunk.start_line}:{chunk.end_line}"
return hashlib.md5(key.encode()).hexdigest()
class ChunkStore:
def __init__(self):
client = chromadb.HttpClient(host=CHROMA_HOST, port=8000)
self._col = client.get_or_create_collection(
name=COLLECTION_NAME,
metadata={"hnsw:space": "cosine"},
)
def upsert(self, chunks: Sequence) -> None:
if not chunks:
return
ids = [_chunk_id(c) for c in chunks]
texts = [c.text for c in chunks]
embeddings = _embed(texts)
metadatas = [
{
"file_path": c.file_path,
"start_line": c.start_line,
"end_line": c.end_line,
"language": c.language,
"node_type": c.node_type,
}
for c in chunks
]
self._col.upsert(ids=ids, embeddings=embeddings, documents=texts, metadatas=metadatas)
def query(self, text: str, n_results: int = 3) -> list[str]:
if self._col.count() == 0:
return []
embeddings = _embed([text])
results = self._col.query(
query_embeddings=embeddings,
n_results=min(n_results, self._col.count()),
include=["documents"],
)
return results["documents"][0]
The upsert method is idempotent — re-indexing a file replaces its chunks rather than duplicating them. The chunk ID is a deterministic hash of file_path:start_line:end_line, so the same chunk always maps to the same Chroma document.
Layer 3: Assembling the FIM Prompt
The FIM prompt has a precise structure. The prefix slot has two sub-parts: retrieved context from the rest of the repo, followed by the current file up to the cursor. The suffix is everything after the cursor.
<fim_prefix>
# Repo context ← top-3 retrieved chunks (~1500 tokens)
# Current file ← lines 0 → cursor (~800 tokens)
<fim_suffix>
# Current file ← cursor → EOF (~400 tokens)
<fim_middle> ← StarCoder2 generates the completion here
Here is server/inference/fim.py:
import os
from dataclasses import dataclass
import httpx
from rag.store import ChunkStore
OLLAMA_HOST = os.getenv("OLLAMA_HOST", "localhost")
FIM_MODEL = os.getenv("FIM_MODEL", "starcoder2:3b")
# StarCoder2 FIM special tokens
FIM_PREFIX = "<fim_prefix>"
FIM_SUFFIX = "<fim_suffix>"
FIM_MIDDLE = "<fim_middle>"
@dataclass
class FIMRequest:
file_path: str
prefix: str # current file content above cursor
suffix: str # current file content below cursor
language: str
def build_fim_prompt(request: FIMRequest, store: ChunkStore) -> str:
# Query the store with the last ~200 chars of prefix as the search query
query = request.prefix[-200:].strip() or request.file_path
context_chunks = store.query(query, n_results=3)
context_block = "\n\n".join(context_chunks)
if context_block:
context_block = f"# Repo context\n{context_block}\n\n# Current file\n"
return (
f"{FIM_PREFIX}"
f"{context_block}"
f"{request.prefix}"
f"{FIM_SUFFIX}"
f"{request.suffix}"
f"{FIM_MIDDLE}"
)
def complete(request: FIMRequest, store: ChunkStore, max_new_tokens: int = 64) -> str:
prompt = build_fim_prompt(request, store)
resp = httpx.post(
f"http://{OLLAMA_HOST}:11434/api/generate",
json={
"model": FIM_MODEL,
"prompt": prompt,
"stream": False,
"options": {
"num_predict": max_new_tokens,
"temperature": 0.1,
"stop": ["\n\n", FIM_PREFIX, FIM_SUFFIX],
},
},
timeout=10,
)
resp.raise_for_status()
return resp.json()["response"]
Two implementation choices worth explaining:
Low temperature (0.1). Autocomplete is not a creative task. You want the most probable continuation, not a diverse sample. High temperature produces hallucinated variable names and incorrect function signatures.
Stop tokens include \n\n. A single blank line is the natural end of a completion. Without this stop token the model continues generating until it hits max_new_tokens, wasting latency and producing over-completion.
Model Selection
Not every developer has the same hardware. Here is the recommended model per tier:
| Model | Size | VRAM | FIM Support | Recommended for |
|---|---|---|---|---|
| StarCoder2-3B | 3B | 4GB | ✅ Native | 8GB GPU or Apple M-series |
| Qwen2.5-Coder-7B | 7B | 8GB | ✅ Native | 16GB GPU |
| DeepSeek-Coder-V2-Lite | 16B | 12GB | ✅ Native | 24GB GPU (best quality) |
| StarCoder2-3B (Q4_K_M) | 3B | 2.5GB | ✅ Native | CPU-only (slow but works) |
Change the model in .env:
FIM_MODEL=qwen2.5-coder:7b
Ollama pulls it on next container start. No code changes required — the FIM special tokens differ between model families but Ollama handles the tokenization automatically.
Layer 4: The VS Code Extension
The extension registers an InlineCompletionItemProvider. VS Code calls it whenever the user pauses typing. The debounce prevents a network round-trip on every keystroke.
Keystroke → Debounce 150ms → Extract context
→ POST /complete {file_path, prefix, suffix, language}
→ Dhi server (FIM prompt → Ollama → completion)
→ Ghost text rendered in editor
→ Tab to accept / Escape to dismiss
Here is extension/src/completion/provider.ts:
import * as vscode from 'vscode';
const SERVER_URL = vscode.workspace
.getConfiguration('dhi')
.get<string>('serverUrl', 'http://localhost:8000');
interface CompletionRequest {
file_path: string;
prefix: string;
suffix: string;
language: string;
}
async function fetchCompletion(req: CompletionRequest): Promise<string | null> {
try {
const res = await fetch(`${SERVER_URL}/complete`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(req),
signal: AbortSignal.timeout(3000),
});
if (!res.ok) return null;
const data = await res.json() as { completion: string };
return data.completion ?? null;
} catch {
return null;
}
}
export class DhiCompletionProvider implements vscode.InlineCompletionItemProvider {
private pending: NodeJS.Timeout | null = null;
async provideInlineCompletionItems(
document: vscode.TextDocument,
position: vscode.Position,
_context: vscode.InlineCompletionContext,
token: vscode.CancellationToken,
): Promise<vscode.InlineCompletionList | null> {
if (this.pending) {
clearTimeout(this.pending);
this.pending = null;
}
const completion = await new Promise<string | null>((resolve) => {
this.pending = setTimeout(async () => {
if (token.isCancellationRequested) { resolve(null); return; }
const offset = document.offsetAt(position);
const text = document.getText();
resolve(await fetchCompletion({
file_path: document.uri.fsPath,
prefix: text.slice(0, offset),
suffix: text.slice(offset),
language: document.languageId,
}));
}, 150);
});
if (!completion || token.isCancellationRequested) return null;
return {
items: [
new vscode.InlineCompletionItem(
completion,
new vscode.Range(position, position),
),
],
};
}
}
Register it in extension.ts:
import * as vscode from 'vscode';
import { DhiCompletionProvider } from './completion/provider';
export function activate(context: vscode.ExtensionContext) {
const provider = new DhiCompletionProvider();
context.subscriptions.push(
vscode.languages.registerInlineCompletionItemProvider(
{ pattern: '**' },
provider,
),
);
}
export function deactivate() {}
And the FastAPI endpoint in server/main.py:
from fastapi import FastAPI
from pydantic import BaseModel
from inference.fim import FIMRequest, complete
from rag.store import ChunkStore
app = FastAPI()
store = ChunkStore()
class CompleteRequest(BaseModel):
file_path: str
prefix: str
suffix: str
language: str
@app.post("/complete")
def complete_endpoint(req: CompleteRequest):
fim_req = FIMRequest(
file_path=req.file_path,
prefix=req.prefix,
suffix=req.suffix,
language=req.language,
)
completion = complete(fim_req, store)
return {"completion": completion}
@app.post("/index")
def index_endpoint(body: dict):
from rag.chunker import chunk_file
chunks = list(chunk_file(body["file_path"]))
store.upsert(chunks)
return {"indexed": len(chunks)}
The /index endpoint is called by a file-watcher in the extension whenever a file is saved. This keeps the vector store in sync with your edits without a full re-index.
Latency in Practice
On an Apple M3 Pro (no external GPU) with StarCoder2-3B via Ollama:
| Scenario | P50 | P95 |
|---|---|---|
| Cold (no Ollama cache) | 380ms | 520ms |
| Warm (model loaded) | 95ms | 145ms |
| Warm + context retrieval | 110ms | 160ms |
The P50 of 95ms warm sits well inside the < 150ms target. Context retrieval adds ~15ms — a small price for the quality improvement from repo-aware completions.
Three things that affect latency more than anything else:
1. Max new tokens. The default is 64. For single-line completions, 32 is enough and nearly halves generation time. Set FIM_MODEL_MAX_TOKENS=32 in .env if you want faster single-line suggestions.
2. Prefix length. Truncate the prefix at ~800 tokens before sending. Longer prefixes increase the prompt processing time quadratically on transformer models.
3. Cold start. The first request after docker compose up is always slow because Ollama loads the model into memory. On M3 Pro this takes ~3 seconds. Subsequent requests hit the warm model cache.
Debounce wait ~80ms
Query embedding ~8ms
Chroma query ~4ms
StarCoder2 (32 tok) ~85ms
Network round-trip ~5ms
─────────────────────────
Total ~110ms P50 ✓ (target: < 150ms)
Indexing Your Repo
The extension calls /index on every file save. To index an existing project on first launch, add a command:
// extension.ts
vscode.commands.registerCommand('dhi.indexWorkspace', async () => {
const files = await vscode.workspace.findFiles(
'**/*.{py,ts,tsx}',
'**/node_modules/**',
);
for (const file of files) {
await fetch(`${SERVER_URL}/index`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ file_path: file.fsPath }),
});
}
vscode.window.showInformationMessage(`Dhi: indexed ${files.length} files`);
});
Run it once with Ctrl+Shift+P → Dhi: Index Workspace. On a medium-sized TypeScript project (~200 files) this takes about 40 seconds and produces ~1,800 chunks.
What We Have So Far
At post-1, Dhi does one thing: it gives you fast, repo-aware FIM autocomplete using entirely open-source components. No API key. No per-token pricing. The full stack fits on a laptop.
The component map so far:
flowchart TB
EXT["VS Code Extension\nInlineCompletionItemProvider"]
API["FastAPI Server\n/complete · /index"]
FIM["fim.py\nFIM prompt builder"]
STORE["store.py\nChroma · cosine search"]
CHUNK["chunker.py\nTree-sitter · semantic chunks"]
OLLAMA["Ollama\nStarCoder2-3B · nomic-embed-text"]
EXT <-->|"HTTP POST /complete"| API
EXT -->|"HTTP POST /index on save"| API
API --> FIM --> STORE --> OLLAMA
API --> CHUNK --> STORE
What's Next
The autocomplete engine queries the vector store at request time. That means the quality of completions depends entirely on the quality of what is in the store. In the next post we go deep on Repo Intelligence — the layer that keeps the store accurate, fast, and in sync with your codebase at all times:
- Full Tree-sitter support for Go, Rust, and Java alongside Python and TypeScript
- LSP call graph: an adjacency list of every function-to-function reference in SQLite
- Hybrid search: nomic vector similarity + BM25 keyword matching, re-ranked with RRF
- Incremental re-index on file save — under 100ms per file even on large repos
- Git-aware indexing: skip
.gitignoreentries, auto-update ongit checkout
The code will be at github.com/sochaty/dhi tag post-2.