On-device, real-time multimodal AI. Have natural voice and vision conversations with an AI that runs entirely on your machine.

Parlor Jarvis is an enhanced fork of Parlor with stronger multilingual capabilities and richer image inputs. You talk, show your camera (or screen, PDF, or video), and it talks back — all locally.

Parlor Jarvis Demo

Video generation in ComfyUI.

Image: AI-generated influencer

Parlor:

Research preview. This is an early experiment. Expect rough edges and bugs.

Parlor vs Parlor Jarvis

Why?

Local models are quickly catching up on both text generation and image understanding, and one of the most exciting breakthroughs is multilingual capability — strong performance across European languages (Spanish, Portuguese, French, …) and Asian languages (Korean, …), not just English. Parlor Jarvis is our take on what a fully on-device, multilingual, multimodal assistant can feel like today: speak any of the supported languages, point your camera, share your screen, drop in a PDF or video, and have a real conversation with a model running entirely on your machine. Enjoy!

Original author's why (from Parlor):

I'm self-hosting a totally free voice AI on my home server to help people learn speaking English. It has hundreds of monthly active users, and I've been thinking about how to keep it free while making it sustainable.

The obvious answer: run everything on-device, eliminating any server cost. Six months ago I needed an RTX 5090 to run just the voice models in real-time.

Google just released a super capable small model that I can run on my M3 Pro in real-time, with vision too! Sure you can't do agentic coding with this, but it is a game-changer for people learning a new language. Imagine a few years from now that people can run this locally on their phones. They can point their camera at objects and talk about them. And this model is multi-lingual, so people can always fallback to their native language if they want. This is essentially what OpenAI demoed a few years ago.

How it works

Browser (mic + camera / screen / PDF / video)
    │
    │  WebSocket (audio PCM + JPEG frames)
    ▼
FastAPI server
    ├── Supergemma 4 E4B via LiteRT-LM (GPU)  →  understands speech + vision
    └── Supertonic TTS (MLX on Mac, ONNX on Linux)  →  speaks back
    │
    │  WebSocket (streamed audio chunks)
    ▼
Browser (playback + transcript)

• Voice Activity Detection in the browser (Silero VAD). Hands-free, no push-to-talk.
• Barge-in. Interrupt the AI mid-sentence by speaking.
• Sentence-level TTS streaming. Audio starts playing before the full response is generated.

Requirements

• Python 3.12+
• Node.js 20+ (for the Next.js frontend)
• macOS with Apple Silicon, or Linux with a supported GPU
• ~3 GB free RAM for the model

Quick start

Parlor Jarvis has two pieces: a FastAPI backend (src/) and a Next.js frontend (web/). In development you run both — Next proxies /ws to FastAPI so the browser talks to a single origin.

git clone https://github.com/typomonster/parlor-jarvis.git
cd parlor-jarvis

Terminal 1 — backend:

# Install uv if you don't have it
curl -LsSf https://astral.sh/uv/install.sh | sh

cd src
uv sync
uv run server.py

Terminal 2 — frontend:

cd web
npm install
npm run dev

Open http://localhost:3000, grant camera and microphone access, and start talking.

Models are downloaded automatically on first run (~2.6 GB for Gemma 4 E2B, plus TTS models).

How the dev proxy works

web/next.config.ts declares a rewrite from /ws → http://localhost:8000/ws, so the browser opens ws://localhost:3000/ws and Next.js proxies the upgrade through to FastAPI. Override the backend target with BACKEND_URL=http://host:port npm run dev if you want to point at a different server.

Configuration

Note: The default configuration is optimized for Korean, even though both the LLM and TTS support other languages natively. You can swap the LLM or TTS models via the env vars below — see Supported models and TTS engines.

Backend (src/)

TTS engines

Parlor Jarvis ships with two interchangeable TTS backends, selected at runtime via TTS_ENGINE:

Force ONNX instead of MLX on macOS with SUPERTONIC_ONNX=1 or KOKORO_ONNX=1. Language is selected automatically from the frontend's active locale (Supertonic only — Kokoro ignores it). Set TTS_ENGINE=kokoro if you want the lighter English-only engine.

Supported models

Switch models by setting both env vars, e.g.:

HF_REPO=litert-community/gemma-4-E2B-it-litert-lm \
HF_FILENAME=gemma-4-E2B-it.litertlm \
uv run server.py

Frontend (web/)

Performance (Apple M3 Pro)

Decode speed: ~83 tokens/sec on GPU (Apple M3 Pro).

Project structure

parlor-jarvis/
├── src/                       # Python backend
│   ├── server.py              # FastAPI WebSocket server + Gemma 4 inference
│   ├── tts.py                 # Platform-aware TTS (MLX on Mac, ONNX on Linux)
│   ├── pyproject.toml         # Python dependencies
│   └── benchmarks/
│       ├── bench.py           # End-to-end WebSocket benchmark
│       └── benchmark_tts.py   # TTS backend comparison
└── web/                       # Next.js frontend (TypeScript + Tailwind + shadcn/ui)
    ├── app/
    │   ├── page.tsx           # Main UI — VAD, camera, audio playback
    │   ├── layout.tsx
    │   └── globals.css        # App styles + Tailwind tokens
    ├── components/ui/         # shadcn/ui components
    └── next.config.ts         # /ws rewrite → FastAPI

Acknowledgments

• Gemma 4 by Google DeepMind
• LiteRT-LM by Google AI Edge
• Supertonic multilingual TTS by Supertone (github)
• Supergemma 4 E4B (abliterated) by Jiunsong
• Kokoro TTS by Hexgrad
• Silero VAD for browser voice activity detection

Our open source contributions

To make Parlor Jarvis possible, we ported and packaged a couple of upstream models for on-device use and shared everything back:

• Supertonic-2 → MLX for fast, native Apple Silicon TTS
  • MLX checkpoint: typomonster/supertonic-2-mlx
  • mlx-audio patch adding Supertonic support: typomonster/mlx-audio
  • Standalone Supertonic + MLX runner: typomonster/supertonic
• Supergemma 4 → LiteRT-LM for on-device multilingual speech + vision
  • LiteRT-LM checkpoint: typomonster/supergemma4-e4b-abliterated-litert-lm

License

Apache 2.0