I didn't set out to build a RAG application. I set out to solve an annoying problem I kept watching happen.
I work as a senior software developer in healthcare technology in Belfast. A big part of that job is understanding what actually slows them down, and figuring out where software can help. Not what's technically impressive what's genuinely useful.
One thing I kept noticing: a lot of time gets spent navigating documents. Not reading them carefully and thoughtfully — just navigating. Ctrl+F for a patient name. Scrolling to find a medication dosage. Hunting for a follow-up instruction buried in paragraph four of page seven of a discharge summary.
It sounds small. Multiply it by every clinical document, every working day, and it adds up fast.
When I started thinking about RAG as a solution, the first thing I did was slow down and think about what "accuracy" means in a clinical setting — because it means something different here than it does in most software contexts.

The Problem With Most AI Demonstrations in Healthcare
Most AI demos in healthcare go like this: upload a document, ask a question, get a fluent confident answer. It looks impressive. The problem is that fluent and confident doesn't mean accurate. Language models are optimised to produce coherent text. Left to their own devices they'll fill gaps, make inferences, and sometimes just invent things — in a way that reads exactly like a real answer.
In most contexts that's an acceptable tradeoff. In a clinical context it isn't. A patient's discharge medications, their follow-up appointments, their documented allergies — these things need to come from the actual document, not from a model's best guess based on what usually appears in documents like this.
So before I wrote a single line of code I made two decisions:

1. Temperature 0. No creativity. Deterministic responses only. The model either finds the answer in the document or it says it doesn't know.

2. Explicit system prompt. The model is told directly: answer only from the provided context. If the answer isn't there, say so clearly. Do not guess.
   These aren't complicated decisions. But they're the right ones for this use case, and I see a lot of healthcare AI demos where nobody made them.

   ---

   How RAG Actually Works (Without the Hype)
   RAG stands for Retrieval-Augmented Generation. The idea is straightforward:
   Instead of asking a language model a question and hoping it knows the answer from training, you first retrieve relevant sections from your actual documents, then ask the model to answer based only on what you retrieved.
   The pipeline looks like this:
   Document (PDF)
   ↓
   Extract text
   ↓
   Split into chunks
   ↓
   Embed each chunk as a vector
   ↓
   Store in a vector database

   ↓ (at query time)

Question
↓
Embed the question as a vector
↓
Find the most similar chunks (semantic search)
↓
Send those chunks + question to the LLM
↓
Get an answer grounded in the document
The key word is grounded. The LLM doesn't know what's in your documents from training — it only knows what you retrieved and passed to it. If the answer isn't in the retrieved chunks, a well-instructed model will tell you that.

Building It: The Decisions That Mattered
Chunking strategy
How you split documents into chunks matters more than most tutorials acknowledge. Too small and you lose context — a medication dosage split from its drug name is useless. Too large and retrieval gets imprecise.
I landed on 500-character chunks with 50-character overlap. The overlap is important — it means the boundary between two chunks always has context from both sides, so you don't lose meaning at the split point.
splitter = RecursiveCharacterTextSplitter(
chunk_size=500,
chunk_overlap=50,
separators=["\n\n", "\n", ". ", " ", ""],
)
The separator hierarchy matters too. We try to split at paragraph boundaries first, then sentence boundaries, then spaces. Splitting mid-sentence is a last resort.
Embedding model
I used OpenAI's text-embedding-3-small. It's fast, cheap, and good enough for document retrieval. For production clinical systems handling complex medical terminology you'd want to evaluate domain-specific embeddings — but for a general-purpose tool this works well.
The system prompt
This is where clinical RAG lives or dies:
system_prompt = (
"You are a helpful assistant that answers questions about clinical documents. "
"Answer only from the provided context. If the answer is not in the context, "
"say so clearly — do not guess or make up information. "
"In a clinical setting, accuracy matters more than completeness."
)
That last sentence — accuracy matters more than completeness — is doing a lot of work. It gives the model permission to say "I don't know" rather than producing a plausible-sounding answer. In my testing, including it made a real difference to the rate of hallucinations on edge cases.

What I Learned Building This
The messy document problem is real. Most RAG tutorials use clean, well-structured PDFs. Clinical documents are not clean or well-structured. Scanned discharge summaries with inconsistent formatting, referral letters with abbreviations that differ between hospitals, guidelines with tables that don't chunk cleanly — all of these degrade retrieval quality. I've started thinking about pre-processing pipelines to handle this better and may add that to the project.
Confidence matters. I added a simple confidence indicator based on how many relevant chunks were retrieved. It's a rough heuristic — more retrieved chunks suggests a more answerable question — but it gives users a signal about how much to trust the response. In a clinical context, knowing when to verify against the source document is as important as the answer itself.
The UI needs to be dead simple. Clinical staff are not developers. If the interface requires any technical knowledge to operate it won't get used. The Streamlit app has three interactions: upload a file, type a question, read the answer. That's it.

The Code
The project is open source on GitHub: github.com/Monika-Sonnadmath/clinical-rag
It has a Python API for developers who want to embed it in their own pipelines, and a Streamlit web app for anyone who just wants to use it directly.
pip install -r requirements.txt
export OPENAI_API_KEY=sk-...
streamlit run app.py
Upload a PDF, ask a question, get an answer. That's the whole thing.

What's Next
A few things I want to add:
• Local LLM support via Ollama — so it works without an API key and keeps documents entirely on-premises. For clinical use cases, data leaving the building is a concern.
• Multi-document querying — ask a question across a folder of documents at once
• Better pre-processing — handling scanned documents with variable quality, which is very much a solved problem in OCR but needs connecting to the retrieval pipeline
If you work in healthcare tech and have thoughts on what would make this more useful, I'd genuinely like to hear from you.