Financial advisors have a dirty secret: they spend almost half their working hours not advising anyone.

The culprit? Compliance documentation. After every client meeting, advisors must document what was discussed, what was recommended, whether those recommendations were suitable, and whether they followed FINRA and SEC rules — all in a format their CRM can ingest.

A 45-minute meeting routinely generates 2 hours of paperwork.

I built an open-source tool that does it in about 3 minutes. Here's exactly how — every architectural decision, every trade-off, and every line of code that matters.

The Problem Is More Specific Than You Think

When I started talking to advisory firms, I expected "meetings take too long" or "we need better CRM software." Instead, every compliance officer said the same thing:

"We're not worried about the notes. We're worried about what's NOT in the notes."

The real pain isn't documentation speed — it's the compliance gap. If a client says "I can't afford to lose this money" and the advisor recommends an aggressive growth fund, that's a FINRA 2111 suitability violation. But if the note-taker (usually the advisor, writing from memory hours later) forgets that quote? No record of the red flag.

This changed my entire system design. It's not a transcription tool with formatting. It's a compliance engine that listens for mismatches.

Architecture

Four-stage pipeline:

Audio → Transcription → Structured Extraction → Compliance Check → CRM Note
         (Whisper)          (Claude via            (Rule engine)    (Formatter)
                             OpenRouter)

Stack: Python/FastAPI + React frontend + Whisper (local) + Claude via OpenRouter

Two key design choices:

1. Whisper runs locally. Advisory meetings contain PII and legally privileged information. Sending audio to third-party APIs isn't optional for most firms — it's a regulatory non-starter.

2. Compliance engine is NOT an LLM. You can't have a probabilistic system making deterministic compliance judgments. The compliance check uses hardcoded rules against structured data. The LLM's job ends at extraction.

The Extraction Prompt

The LLM receives raw transcript text and returns structured JSON. The critical field is risk_signals:

"risk_signals": [
    {
      "signal": "What was said",
      "severity": "low|medium|high",
      "context": "Surrounding context"
    }
  ]

Most meeting note-takers extract topics and action items. This system specifically hunts for phrases indicating compliance risk — "I can't afford to lose this money" (risk tolerance), "my wife doesn't know about this account" (documentation issue), "just put it wherever you think is best" (discretionary authority concern).

Temperature is set to 0.1. When extracting compliance-relevant data, you don't want creative interpretation.

The Compliance Engine

The simplest code in the project. That's the point.

The suitability check cross-references risk signals against product recommendations:

risk_averse_keywords = ["can't afford to lose", "conservative", "safe",
                        "no risk", "preserve capital", "nervous"]
aggressive_products = ["growth fund", "equity", "crypto", "leveraged",
                       "options", "futures", "speculative"]

if has_risk_averse and has_aggressive:
    status = "RED"
    flags.append("SUITABILITY CONCERN: Risk-averse language detected "
                 "alongside aggressive product recommendations")

Is this keyword matching? Yes. Is it sophisticated? Not really. Is it exactly what compliance officers need? Absolutely.

Output is a traffic-light system: GREEN / YELLOW / RED. This maps directly to how compliance departments already think.

The Reg BI check catches the most common examination finding:

has_alternatives = len(products) > len(recommended)
if not has_alternatives:
    status = "YELLOW"
    flags.append("Product recommended without documented alternatives")

Not glamorous. Prevents six-figure fines.

Why Open Source?

The code isn't the competitive advantage. Any competent developer can chain Whisper → LLM → rule engine.

The value is in:

1. Knowing which problem to solve. Features came from dozens of conversations with compliance officers, not from a spec.
2. Production hardening. SOC 2, on-prem deployment, CRM integrations, SSO, audit logging.
3. Ongoing support. Compliance rules change. FINRA updates regulations.

That's what I do. I automate chaos — messy, manual, regulated workflows that require domain expertise on top of engineering skill.

Full codebase: github.com/archit-akg13/advisor-meeting-notetaker

I'm Archit Mittal. I build AI automation for financial services. If your team spends more time on paperwork than on the work itself, I can fix that → architmittal.com

Follow me on Twitter @automate_archit for daily AI automation tips