TL;DR: Phase 1 of a from-scratch RAG app — Spring AI, pgvector, local Ollama — ends with a working pipeline and two failures that look identical from the outside but have nothing to do with each other. One was a chunking bug. The other was a 3B model running out of brain. Here's how I told them apart.

Why build this

I'm finishing my apprenticeship as a IT specialist in application development and wanted a portfolio project that's more than another CRUD app. Kenning is a document-chat tool: upload a file, ask questions about it, get answers with sources attached. Standard RAG (Retrieval-Augmented Generation), but built end to end by hand instead of stitched together from a tutorial.

Phase 0 was infrastructure: Docker Compose with pgvector/pgvector:pg16 and ollama/ollama, a Spring Boot scaffold, an Angular scaffold. Phase 1's job was narrower and more important: prove the actual RAG loop works — upload one document, ask one question, get a real answer with the source attached. No login, no UI polish, no multi-document handling. Just: does this architecture actually do the thing.

The stack

• Java 21, Spring Boot 4.1.0, Spring AI
• Angular (frontend, mostly untouched in this phase)
• PostgreSQL + pgvector as the vector store
• Ollama, running locally: nomic-embed-text for embeddings and llama3.2:3b for chat

The pipeline: upload → Apache Tika extracts text → TokenTextSplitter chunks it → Spring AI's Ollama embedding client turns each chunk into a vector → pgvector stores it → a question gets embedded the same way → similarity search pulls the closest chunks → those get stuffed into a ChatClient call alongside the question → the model answers, and I attach the source chunks it used.

That's the theory. None of it survived contact with reality without a fight.

Mistake #1: naming my own entity Document

Spring AI ships its own Document class for representing a chunk of text plus metadata. I also wanted an entity called Document for "a file the user uploaded." Same name, two completely different things, one annoying import ambiguity every time autocomplete guessed wrong. Renamed mine to SourceDocument and moved on.

Mistake #2: two Ollamas, one port

I ran qwen2.5-coder:14b natively on Windows for my coding assistant to test out local models and forgot about it. Docker Compose also wants port 11434 for the Ollama container that's supposed to serve Kenning. Two processes, one port, predictable result. The fix was trivial — stop the native Windows process before starting the container — but the error message gave zero hint that this was the cause. Worth remembering if you run a local AI coding tool and an Ollama-based app on the same machine.

Mistake #3 (not really a mistake): the GPU that does nothing

I have an AMD RX 6700 XT with 12 GB of VRAM sitting in this machine, doing nothing for local inference. Rather than assume, I checked:

docker exec -it ollama ollama ps

NAME                       ID              SIZE      PROCESSOR    CONTEXT    UNTIL
llama3.2:3b                a80c4f17acd5    2.6 GB    100% CPU     4096       4 minutes from now
nomic-embed-text:latest    0a109f422b47    376 MB    100% CPU     2048       4 minutes from now

100% CPU, confirmed. As far as I can tell, the issue is that ROCm needs /dev/kfd exposed to the container, and WSL2 doesn't expose it — so GPU acceleration for Ollama running inside Docker on WSL2 seems to be a dead end with this setup.

One thing I haven't tried yet: running Ollama natively on Windows instead of inside WSL2/Docker. Native Windows Ollama has its own path to the GPU drivers that doesn't go through WSL2's passthrough limitations, so that's probably the more realistic way to actually use the 6700 XT — I just haven't switched Kenning over to test it. For now: CPU-only for local dev, on a machine where RAM is already sitting at 96% usage before the model even loads.

For the eventual public demo, the plan is to be upfront about it — a small local model on a GPU-less VPS will be slow and occasionally wrong, and I'll say so, with an option to plug in your own API key if you want better answers.

The real test: asking it about myself

Once the loop worked mechanically, I needed a document with facts I could verify myself. So I wrote a short bio — what Kenning is, what stack it uses, what I'm into outside of code, including one line: "He enjoys mechanical keyboards and is interested in BMW cars."

Question 1: "What embedding model does this project use?"

The answer is one unambiguous sentence in the document: nomic-embed-text for embeddings, llama3.2:3b for chat. The chunk had actually been retrieved — the full document came back in sources — so this wasn't a missing-context problem. Here's what the model actually answered:

"I don't know which embedding model this project specifically uses. The text mentions that Ollama provides two models: nomic-embed-text for generating embeddings, but it does not specify which one is used by Kenning Project."

Read that twice. The model opens by saying it doesn't know, then in the very next sentence names the correct embedding model — nomic-embed-text for generating embeddings — and then closes by claiming the project doesn't specify which one it uses. It has the right answer sitting in its own explanation and still doesn't commit to it.

My read on it: that original sentence names two models in one clause, each bound to a different job ("X for embeddings, Y for chat"), and a 3B model running on CPU can apparently retrieve the right name but not lock it in as a confident final answer. I haven't tested this side by side against a bigger model yet, but based on what I'm seeing, I'd assume something like an 8B+ model would commit to the answer instead of second-guessing itself — that's the next experiment, not something I've actually confirmed.

Question 2: "What car brand do I like?"

Zero source matches. No chunk even cleared the similarity threshold (0.5) to make it into the prompt. Lowering the threshold to 0.3 surfaced it — at an actual similarity score of 0.46, just under the original cutoff.

This looks like the same kind of failure as Question 1, but I don't think it is. My test document was short enough (2,063 characters) to stay as a single chunk. That one chunk covers Spring AI, Tika, async processing, OAuth2 plans — and, almost as an afterthought in the last sentence, BMW. My working theory: embedding a chunk that mixed produces a vector that's diluted across five unrelated topics, so a focused query like "what car brand" scores lower against it than it would against a clean, topic-specific chunk. I haven't actually re-run it with smaller, topic-coherent chunks to confirm yet — but that's the fix I'd try next, and my guess is it would clear the threshold comfortably.

Two questions, two failures that looked identical from where I was sitting ("the bot doesn't know basic facts about me") but, as far as I can tell, have different causes underneath. One I'd expect a bigger model to fix. The other I'd expect better chunking to fix. Telling them apart — even just as working theories — was the most useful thing to come out of this phase.

Where Phase 1 lands

A document goes in, a question goes in, an answer comes back with its source chunk attached — all on a fully local stack, no API costs, end to end. It's slow, it's occasionally wrong in the specific way small models are wrong, and the chunking is still naive. But the architecture holds up, and proving that was the actual point of this phase.

If you've actually debugged RAG pipelines at this level, I'd genuinely like to know whether the two working theories above hold up — the attribute-binding explanation for the embedding-model question, and the chunk-dilution explanation for the BMW question. I'm reasoning from what I observed here, not from having traced either one to the bottom, so if you've got more experience with this and either guess is wrong, half right, or missing something obvious, I'd love to hear it.

Phase 2 is auth (Google OAuth2), a real upload UI, and multi-document support per user. I'll write that one up once it's running.