惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

美团技术团队
W
WeLiveSecurity
Stack Overflow Blog
Stack Overflow Blog
L
LangChain Blog
S
SegmentFault 最新的问题
Apple Machine Learning Research
Apple Machine Learning Research
Google DeepMind News
Google DeepMind News
F
Full Disclosure
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报
The Register - Security
The Register - Security
G
Google Developers Blog
C
Check Point Blog
GbyAI
GbyAI
A
About on SuperTechFans
V
Vulnerabilities – Threatpost
T
The Blog of Author Tim Ferriss
T
Tor Project blog
AWS News Blog
AWS News Blog
Cyberwarzone
Cyberwarzone
C
CERT Recently Published Vulnerability Notes
MongoDB | Blog
MongoDB | Blog
Latest news
Latest news
aimingoo的专栏
aimingoo的专栏
U
Unit 42
Y
Y Combinator Blog
P
Privacy International News Feed
Cisco Talos Blog
Cisco Talos Blog
S
Securelist
S
Schneier on Security
雷峰网
雷峰网
cs.CV updates on arXiv.org
cs.CV updates on arXiv.org
Attack and Defense Labs
Attack and Defense Labs
P
Proofpoint News Feed
C
Cisco Blogs
Webroot Blog
Webroot Blog
T
Troy Hunt's Blog
Google Online Security Blog
Google Online Security Blog
月光博客
月光博客
P
Privacy & Cybersecurity Law Blog
Security Archives - TechRepublic
Security Archives - TechRepublic
罗磊的独立博客
Cloudbric
Cloudbric
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
Recent Commits to openclaw:main
Recent Commits to openclaw:main
Application and Cybersecurity Blog
Application and Cybersecurity Blog
Hacker News: Ask HN
Hacker News: Ask HN
H
Hackread – Cybersecurity News, Data Breaches, AI and More
博客园 - 司徒正美
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
Microsoft Security Blog
Microsoft Security Blog

DEV Community

Authentication Security Deep Dive: From Brute Force to Salted Hashing (With Java Examples) Why AI Systems Don’t Fail — They Drift Spilling beans for how i learn for exam😁"Reinforcement Learning Cheat Sheet" I Replaced Chrome with Safari for AI Browser Automation. Here's What Broke (and What Finally Worked) How Python Borrows Other People's Work The $40 Architecture: Processing 1 Billion API Requests with 99.99% Uptime Vibe Coding: A Workflow Guide (From Zero to SaaS) Most webhook security guides protect the wrong side. The scary part is delivery. Headless CMS for TanStack Start: Build a Blog with Cosmic EU Age Verification App "Hacked in 2 Minutes" — What Actually Happened Comfy Cloud’s delete function does not actually remove files Running AI Models on GPU Cloud Servers: A Beginner Guide Event-driven media intelligence with AWS Step Functions and Bedrock I scored 500 AI prompts across 8 quality dimensions — here's what broke How to Call Google Gemini API from Next.js (Free Tier, No Backend Needed) The Portal Protocol: Reclaiming Human Connection in the Age of AI How to Fix Your Team's Scattered Knowledge Problem With a Self-Hosted Forum Intro to tc Cloud Functors: A Graph-First Mental Model for the Modern Cloud Designing Multi-Tenant Backends With Both Ownership and Team Access I Built a Neumorphic CSS Library with 77+ Components — Here's What I Learned PostgreSQL Performance Optimization: Why Connection Pooling Is Critical at Scale Cómo construí un SaaS multi-rubro para gestionar expensas en Argentina con FastAPI + Vue 3 🚀 I Built an Ethical Hacking Scanner Tool – Open Source Project I Replaced /usage and /context in Claude Code With a Single Statusline A Pythonic Way to Handle Emails (IMAP/SMTP) with Auto-Discovery and AI-Ready Design I Collected 8.9 Million Polymarket Price Points — Here's What I Found About How Markets Really Move EcoTrack AI — Carbon Footprint Tracker & Dashboard Everyone's Using AI. No One Agrees How. 5 self-hosted ebook managers worth trying in 2026 Building Your First AI Agent with LangChain: From Chatbot to Autonomous Assistant Common SOC 2 Failures (Real World) Stop Vibe-Checking Your AI App: A Practical Guide to Evals How to Use SonarQube and SonarScanner Locally to Level Up Your Code Quality Your Next To-Do App Is Dead — I Replaced Mine with an OpenClaw AI Sign a Nostr event in 60 lines of Python using coincurve — no nostr-sdk, no nbxplorer, no rust toolchain ITGC Audit Explained Like You’re in Big 4 Patch Tuesday abril 2026: Microsoft parcha 163 vulnerabilidades y un zero-day en SharePoint Stop scraping everything: a better way to track competitor price changes Listing on MCPize + the Official MCP Registry while routing payments OUTSIDE the marketplace — how I kept 100% of my x402 revenue Building an AI-Powered Risk Intelligence System Using Serverless Architecture Why We Ripped Function Overloading Out of Our AI Toolchain Testing AI-Generated Code: How to Actually Know If It Works SaaS Churn Is Killing Your Business. Here Is What to Do About It (Without a Support Team) The Speed of AI Is No Longer Linear - And Self-Improving Models Are Why How to Implement RBAC for MCP Tools: A Practical Guide for Engineering Teams From Standard Quote to Persuasive Proposal: AI Automation for Arborists I built a CLI that scaffolds complete multi-tenant SaaS apps Axios CVE-2025–62718: The Silent SSRF Bug That Could Be Hiding in Your Node.js App Right Now The dashboard that ended our friendship Data Pipelines Explained Simply (and How to Build Them with Python) The Hidden Cost of AI Systems Nobody Talks About. undefined vs undeclared, and how typeof behaves Switching from file-based jobs to NATS/Kafka in Rust without changing code io_uring Adventures: Rust Servers That Love Syscalls Why Agentic AI is Killing the Traditional Database The POUR principles of web accessibility for developers and designers Quantum Neural Network 3D — A Deep Dive into Interactive WebGL Visualization How To Install Caveman In Codex On macOS And Windows Automation Pipeline Reliability: Why Your Workflow Breaks When Nobody Is Watching I Built an 'Open World' AI Coding Agent — It Works From ANY Folder From Freelancing to Product: A Tech Service Company's SaaS Transformation China's AI Giants: Adding Tencent Hunyuan & ByteDance Doubao to AI University (74 Providers) On the Vibe Coders and Their Lies clerk: Auto-Summarize Your Claude Code Sessions AI Weekly — 2026/04/10–04/17 | The Model Lockdown Is Here, but the Toolchain Is the Real Battleground AI 週報 — 2026/04/10–2026/04/17 模型封鎖潮來了,但工具鏈才是真戰場 Maybe this is how Open-Source apps are born... 🚀 Fine-Tune LLMs with LoRA and QLoRA: 2026 Guide tRPC v11 + Next.js App Router: End-to-End Type Safety Without the Boilerplate ShadCN UI in 2026: Why I Stopped Installing Component Libraries and Started Owning My Components SaaS Billing in React Server Components: Stripe + Supabase Without a Single `useEffect` Join our DEV Weekend Challenge — $1,000 in Prizes Across TEN winners! Submissions Due April 20 at 6:59 AM UTC. Implementing FSRS Spaced Repetition in Flutter + Supabase — Adding Memory Science to an AI Learning App "I Texted My Localhost From the Train — Claude Code Fixed the Bug Before I Got Home" I Built a Sales Prep AI and It Went Deeper Than Expected Design to Code #2: One JSON, Eleven Outputs Solving the 100M-Row Problem: A Summary Table Pattern for High-Volume Push Notification Logs Flutter Web With Wasm: What Actually Changes For Developers I Built 50 Royalty-Free Soundtracks for My Side Project in a Weekend Using AI Music Generation The Vibe Coding Security Checklist: 7 Things to Check Before You Ship Stop Letting Googlebot Guess Fix Your React App's SEO Right Desconstruindo o Streaming do LinkedIn: Como Criar um Engine de Extração de Vídeo de Alta Performance com HLS e FFmpeg (EDA Part-1) EDA (Exploratory Data Analysis) Explained With Real Life — Why Looking at Your Data Is the Most Important Step in Machine Learning Brand Relationship Management at Scale: Our 4-Touch Outreach System for 200+ Brands Why String.fromEnvironment() Might Return an Empty String in Dart JGuardrails 1.0.0 — Hardening Java LLM Apps Against Jailbreaks, Toxicity, and Prompt Injection Plan and Schedule a Full Week of Threads Content From One Claude Conversation Coding Cat Oran Ep3, Five Tables Changed Everything Updated: BFF Pattern I'm done watching freelancers get buried by 200 proposals. So I'm building the alternative. This is my first post BFS Algorithm in Java Step by Step Tutorial with Examples Tracking LLM Pricing Monthly: An Open Dataset for 22 AI Models How We Measure Content ROI on a Comparison Site: Revenue Attribution Without Perfect Data Introducing Nova AI Ops: The AI-Native Operating System for SRE Teams I built a free desktop video downloader for Windows — Grabbit How Talkie OCR Helps Vision-Impaired & Dyslexic Users Read the World Around Them VRCFaceTracking安装和iPhone面捕配置教程,有bug Even CrowdStrike Can't See Your Agents The Automation Gold Rush: What n8n Workflows and Claude Are Opening Up for Developers Right Now
Moving Beyond the Black Box: How I Built a Real-Time Voice Fitness Coach using Next.js 15, Convex, & Vapi.ai
Sanya Prabha · 2026-05-21 · via DEV Community

Moving Beyond the Black Box: How I Built a Real-Time Voice Fitness Coach using Next.js 15, Convex, & Vapi.ai

I've used a lot of fitness apps. I've followed their calorie numbers, done their workout splits, and trusted their recommendations - all without ever understanding where any of it came from. The number just appeared, authoritative and unexplained, and I was supposed to trust it.

Eventually, I stopped. Not because the numbers were wrong. Because I had no reason to believe they were right.

That frustration became FitExplain - a full-stack web application that generates personalized fitness and nutrition plans through a voice conversation, and then shows you exactly how every number was calculated. No black box. No mystery algorithm. Just metabolic science, made visible.

This post is a complete technical breakdown of how I built it.


🧠 The Core Problem I Was Solving

Most fitness apps share a quiet design flaw: they treat the user like they can't handle the truth.

You open the app, enter your weight and height, tap through a few goal screens, and get back a daily calorie target and a workout plan. The numbers look precise. They come with decimal points. But if you ask why 1,940 calories - why not 1,800, why not 2,100 - the app has no answer. It never does.

This isn't accidental. It's a design assumption: that authority is enough, that users will follow a number simply because the app said so.

Behavioral research consistently shows this assumption is wrong. People follow health guidance longer and more faithfully when they understand the reasoning behind it. Unexplained AI output, however accurate, produces shallow trust - and shallow trust doesn't survive the friction of real life.

FitExplain was built around the opposite assumption: show your work.


🏗️ Architecture Overview: The Triple-Cloud Handshake

Before diving into each piece, here's how the system fits together. I call it the Triple-Cloud Handshake:

User's Browser
    │
    ├──── speaks to ──────► Vapi.ai (Voice AI)
    │                           │
    │                           └── webhook ──► Convex Backend
    │                                               │
    ├──── subscribes to ─────────────────────────► Convex (Reactive DB)
    │                                               │
    └──── authenticated by ──► Clerk ──webhook──►  Convex

Enter fullscreen mode Exit fullscreen mode

Four independent cloud services, each best-in-class for its specific role:

Layer Technology Role
Frontend Next.js 15 + Vercel UI, routing, dashboard
Voice AI Vapi.ai Conversational data collection
Backend + DB Convex Persistence, AI orchestration, real-time push
Auth Clerk User identity, session tokens

The browser talks to Vapi for voice and to Convex for data. Clerk and Vapi each communicate with Convex via authenticated server-side webhooks — invisible to the user, but the backbone of the system.


🎙️ Layer 1: Vapi.ai — Voice as the Intake Form

The first question I had to answer was: why voice at all?

Most fitness apps use forms. Forms are fine — until you realize what they actually produce. A dropdown asking "How active are you?" with options like "Sedentary," "Lightly Active," and "Moderately Active" doesn't capture useful data. Users don't know what "moderately active" means in caloric terms, so they pick whatever sounds right and end up with a generic output.

Voice is different. A conversation can ask follow-up questions. It can confirm ambiguous answers. It can be designed to elicit specific, structured information — in a format that feels completely natural to the user.

The Vapi Workflow: 10 Nodes, 8 Parameters

I built a ten-node conversation workflow in Vapi that collects exactly eight parameters before allowing the session to close:

  1. Age - integer
  2. Current weight - numeric, with unit confirmation
  3. Height — numeric, with unit confirmation
  4. Existing injuries - free text, normalized
  5. Primary fitness goal - categorized (fat loss / muscle gain / endurance / general fitness)
  6. Preferred weekly training sessions - integer (1–7)
  7. Current fitness level - categorized (beginner / intermediate / advanced)
  8. Dietary restrictions or allergies - free text, normalized

The workflow uses guard conditions - Vapi will not advance to plan generation unless every required parameter has been captured and confirmed. If a user says something ambiguous like "I work out sometimes," the workflow probes: "About how many days a week would you say?" The session runs between 30 and 77 seconds across all tested users.

The Race Condition I Spent an Hour Debugging

Here's something that cost me a full debugging session: Vapi's call.ended webhook fires before the call analysis is complete. I had built my Convex action to extract parameters from the analysis object - which wasn't populated yet when the webhook arrived.

The fix was to listen for end-of-call-report instead of call.ended. The report event fires after analysis completes and includes the full structured output. A one-line change, but I only found it after reading the Vapi docs three times and adding console logs to every webhook handler.

// convex/http.ts - webhook handler
export const vapiWebhook = httpAction(async (ctx, request) => {
  const payload = await request.json();

  // Use end-of-call-report, NOT call.ended
  if (payload.message?.type === "end-of-call-report") {
    const analysis = payload.message.analysis;
    const structuredData = analysis?.structuredData;

    if (structuredData) {
      await ctx.runAction(internal.fitplan.generatePlan, {
        callId: payload.message.call.id,
        userData: structuredData,
      });
    }
  }

  return new Response("OK", { status: 200 });
});

Enter fullscreen mode Exit fullscreen mode


⚡ Layer 2: Convex — The Reactive Backend

This was the most interesting architectural decision in the project, and the one I'd recommend most strongly to other developers.

Why Not Just Use a REST API?

The standard approach would be: Vapi webhook hits an API route → API route calls Gemini → API route writes to a database → user refreshes the page to see results.

That works. But it means the user has to do something to see their plan. They have to know to refresh. The experience has a gap in it.

Convex eliminates the gap entirely.

How Convex Reactivity Works

In Convex, queries are subscriptions. When the frontend runs a query through the Convex React SDK, it opens a persistent WebSocket connection. Convex tracks which database documents that query depends on. The moment any of those documents change — from any mutation, anywhere in the system - Convex re-executes the query and pushes the updated result to all subscribed clients.

The developer writes a query function. Convex handles subscription, invalidation, and delivery automatically.

// convex/fitplans.ts
export const getUserPlan = query({
  args: { userId: v.string() },
  handler: async (ctx, { userId }) => {
    return await ctx.db
      .query("fitPlans")
      .withIndex("by_user", (q) => q.eq("userId", userId))
      .order("desc")
      .first();
  },
});

Enter fullscreen mode Exit fullscreen mode

// app/dashboard/page.tsx — React component
const plan = useQuery(api.fitplans.getUserPlan, { userId: user.id });

// 'plan' updates automatically when Convex pushes a new value.
// No polling. No useEffect with fetch. No page reload.

Enter fullscreen mode Exit fullscreen mode

For FitExplain, this meant: the moment the Vapi webhook triggers a plan creation mutation in Convex, the user's dashboard re-renders with the new plan. Latency between database write and screen update is WebSocket round-trip time plus Convex re-execution time — under 200ms combined.

The Schema: Keeping Gemini Honest

One subtle but important decision: I defined a strict Convex schema for fitness plans, then validated all Gemini output against it before writing to the database. This caught a bug that took me a while to find — Gemini was returning "2200" (string) instead of 2200 (integer) for calorie values. The Convex validator rejected it, which surfaced the issue immediately rather than letting corrupt data reach the frontend.

// convex/schema.ts
fitPlans: defineTable({
  userId: v.string(),
  callId: v.string(),
  createdAt: v.number(),
  userProfile: v.object({
    age: v.number(),
    weightKg: v.number(),
    heightCm: v.number(),
    goal: v.string(),
    fitnessLevel: v.string(),
    weeklySessionTarget: v.number(),
    injuries: v.string(),
    dietaryRestrictions: v.string(),
  }),
  metabolics: v.object({
    bmr: v.number(),         // Mifflin-St Jeor result
    tdee: v.number(),        // BMR × activity multiplier
    targetCalories: v.number(), // TDEE adjusted for goal
    proteinGrams: v.number(),
    carbGrams: v.number(),
    fatGrams: v.number(),
  }),
  workoutPlan: v.array(v.object({
    day: v.string(),
    focus: v.string(),
    exercises: v.array(v.object({
      name: v.string(),
      sets: v.number(),
      reps: v.string(),
      rest: v.string(),
    })),
  })),
  mealPlan: v.array(v.object({
    meal: v.string(),
    foods: v.array(v.string()),
    calories: v.number(),
  })),
}).index("by_user", ["userId"]),

Enter fullscreen mode Exit fullscreen mode


🤖 Layer 3: Google Gemini 1.5 Flash - Structured AI Output

Most LLM integrations treat the model as a conversational partner. FitExplain treats Gemini as a structured data generator.

The distinction matters. A conversational response ("Here's your plan! On Monday, you should do...") cannot be stored cleanly in a database or rendered in a structured dashboard UI. Schema-compliant JSON can.

The Prompt Engineering

The system prompt does three things:

  1. Instructs Gemini to calculate BMR using Mifflin-St Jeor explicitly
  2. Instructs Gemini to derive TDEE using the standard activity multiplier table
  3. Requires output as valid JSON matching a predefined schema - no prose, no markdown, no preamble
// convex/fitplan.ts - Gemini prompt construction
const systemPrompt = `You are a fitness and nutrition AI that generates structured plans.
You MUST respond with ONLY valid JSON. No markdown. No explanation. No preamble.

Calculate BMR using Mifflin-St Jeor:
- Men: (10 × weight_kg) + (6.25 × height_cm) - (5 × age) + 5
- Women: (10 × weight_kg) + (6.25 × height_cm) - (5 × age) - 161

Apply TDEE multiplier:
- Sedentary (1-2 days/week): BMR × 1.2
- Lightly active (3 days/week): BMR × 1.375
- Moderately active (4-5 days/week): BMR × 1.55
- Very active (6-7 days/week): BMR × 1.725

Adjust target calories for goal:
- Fat loss: TDEE - 500
- Muscle gain: TDEE + 300
- Endurance/General: TDEE

Output JSON matching this exact schema: ${JSON.stringify(planSchema)}`;

Enter fullscreen mode Exit fullscreen mode

The apiVersion Bug

The Gemini Node.js SDK was returning HTTP 404 errors on every call. The fix was non-obvious: I needed to pass apiVersion: 'v1' explicitly when initializing the client. The default was pointing to a preview endpoint that wasn't available in my region.

import { GoogleGenerativeAI } from "@google/generative-ai";

const genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY!, {
  apiVersion: 'v1', // Required — default causes 404 in some regions
});

Enter fullscreen mode Exit fullscreen mode

One line. Forty-five minutes to find it.


🔐 Layer 4: Clerk - Authentication Done Right

I chose Clerk over NextAuth for one reason: its webhook system integrates cleanly with Convex. When a user signs up, Clerk fires a user.created webhook to Convex, which creates a user document in the database. Session tokens from Clerk are verified in every Convex query and mutation, so user data is always scoped correctly.

// convex/http.ts — Clerk webhook
export const clerkWebhook = httpAction(async (ctx, request) => {
  const event = await validateClerkWebhook(request);

  if (event.type === "user.created") {
    await ctx.runMutation(internal.users.createUser, {
      clerkId: event.data.id,
      email: event.data.email_addresses[0].email_address,
      name: `${event.data.first_name} ${event.data.last_name}`,
    });
  }

  return new Response("OK", { status: 200 });
});

Enter fullscreen mode Exit fullscreen mode


📊 The XAI Part: Making the Math Visible

The Explainable AI component of FitExplain isn't a separate feature - it's built into the dashboard display. Every number shown to the user has a derivation trace:

Your Daily Calorie Target: 1,940 kcal

How we calculated this:
├── BMR (Mifflin-St Jeor): 1,847 kcal
│   └── (10 × 75kg) + (6.25 × 178cm) - (5 × 24) + 5
├── TDEE (activity multiplier): 2,440 kcal
│   └── BMR × 1.325 (4 sessions/week = lightly-to-moderately active)
└── Fat loss adjustment: -500 kcal
    └── Standard 500kcal deficit targets ~0.5kg/week loss

Enter fullscreen mode Exit fullscreen mode

This is the core design principle: explainability as engineering input, not post-hoc annotation. I didn't generate a plan and then try to explain it afterward. I started from the explanation — the formula, the multiplier table, the adjustment logic — and used Gemini to instantiate that explanation for each individual user's parameters.

The AI doesn't replace the science. It applies it.


🧪 Testing Results

I ran 13 live voice sessions across users with different profiles:

Metric Result
Voice-to-dashboard latency 5–11 seconds
Convex cached query performance < 5ms
Sessions completed without error 13/13
Schema validation failures 2 (caught before DB write)
Contextually differentiated plans 13/13

The two schema validation failures were both the string/integer type mismatch on calorie values — Gemini occasionally returns numeric values as strings. I added a normalization step in the Convex action that coerces all numeric fields before validation, which resolved both failures.


🚀 The Full Tech Stack

Frontend:     Next.js 15 (App Router), TypeScript, Tailwind CSS
Deployment:   Vercel (automatic from GitHub)
Voice AI:     Vapi.ai (10-node workflow, guard-conditioned)
Database:     Convex (reactive, serverless, TypeScript-native)
AI Model:     Google Gemini 1.5 Flash (structured JSON output)
Auth:         Clerk (webhooks + session verification)

Enter fullscreen mode Exit fullscreen mode


💡 What I'd Do Differently

1. Schema validation from day one. I added the Convex schema validator after encountering the string/integer bug. I should have defined it before writing any Gemini integration code. Strict schema-first development would have caught that issue in local testing rather than in a live session.

2. Vapi workflow versioning. The Vapi workflow is configured in their dashboard UI, which makes version control awkward. I'd build a script to export and commit the workflow JSON alongside the codebase from the start.

3. Gemini output temperature. I used the default temperature (1.0) throughout development. In retrospect, setting temperature to 0.3–0.5 for structured JSON generation would have produced more consistent outputs and fewer edge cases in the schema normalization layer.


🔭 What's Next

The current system generates a plan once per voice session. The obvious next step is longitudinal tracking — storing multiple plans over time, detecting progress, and adjusting recommendations based on what's changed. Convex's reactive queries make this architecturally straightforward; it's primarily a UI and prompt engineering problem.

Other directions I want to explore:

  • Wearable integration - pulling actual activity data from Apple Health or Garmin rather than relying on self-reported activity levels
  • Plan revision via voice - letting users update specific parameters (new injury, changed schedule) without redoing the full intake conversation
  • Nutrition logging - tracking actual meals against the generated plan, with Convex reactively updating macro targets based on what's been logged

Wrapping Up

The thing I kept thinking about while building FitExplain is that the transparency problem in fitness apps isn't really a hard technical problem. It's a design choice. Someone decided that showing users the formula would confuse them, or slow them down, or make the app feel less magical.

I think that's wrong. Showing the formula is what builds real trust — the kind that survives a bad week and a missed workout, instead of evaporating the moment the user questions the number.

With cloud-native tools available today — Vapi for voice, Convex for reactive data, Gemini for structured AI generation — you can build a system that is both intelligent and transparent without trading one for the other.

That's what FitExplain does. And it's what I think a lot more fitness software should do.


The full project is documented in my final year report at Manipal University Jaipur. If you have questions about any specific part of the implementation — the Vapi workflow design, the Convex schema, the Gemini prompt structure — drop them in the comments.


Tags: #ai #gemini #frontend #database