Captures your screen → Analyzes with Gemma 4 → Builds a searchable AI memory
100% local. 100% private. Zero cloud dependencies.

Features · Gemma 4 Deep Dive · Quick Start · Architecture · Agent Platform · MCP · API

Agents

💬 Chat in Action — Ask anything about your screen history

Microsoft showed the world wants screen-aware AI with Recall. But Recall stores data in plaintext, sends telemetry, and was met with massive privacy backlash. ScreenMind is the open-source, privacy-first alternative — every screenshot analyzed, every insight generated, every search result — all computed locally using Gemma 4's multimodal capabilities.

It's not just a screen recorder. It's an AI memory you can talk to, search through, and build automations on top of.

✨ Features

🧠 Core Intelligence

• 📸 Smart Capture — Content-change detection, not a fixed timer. Captures when your screen actually changes.
• 🔬 Gemma 4 Vision Analysis — Every screenshot analyzed: app detection, activity categorization, mood, scene description, spatial layout regions.
• 🔍 Hybrid Search — Semantic embeddings (MiniLM) + FTS5 keyword search. Find anything by meaning, not just keywords.
• 💬 Chat with Memory — Conversational RAG with follow-up support. Ask "what did Alex say on Discord?" → get the actual message.
• 🧠 Model Hub — In-app model download with live progress tracking. Chat and Summary are locked with witty brain animations until the model is ready — then auto-unlock. No terminal needed.
• 🎙️ Voice Memos — Hold Ctrl+Shift+V → Gemma 4's native audio encoder transcribes. Screenshot captured alongside.
• 🎤 Meeting Transcription — Auto-detects Zoom/Teams/Meet, records audio, transcribes, generates structured summaries.
• 📊 Analytics Dashboard — Category breakdown, top apps, hourly heatmap, meeting stats, focus metrics.
• ⏪ Day Rewind — Timelapse playback of your entire day with play/pause/scrub/speed controls.

⚡ Performance

• Three Analysis Modes — Accurate (~76s, deep thinking + layout), Balanced (~40s, thinking), or Fast (~12s, no thinking). You choose.
• Per-App pHash Cache — 3-tier caching with app-aware staleness. Communication apps refresh faster than IDEs. Significantly fewer inference calls.
• Chat-First GPU Priority — Chat cancels in-flight analysis instantly. GPU freed in <1s.
• Auto-Pause Heavy Apps — Games, video editors, 3D software detected → capture pauses automatically.

🔒 Privacy & Security

• 100% Local — All data stays on your machine. Zero network calls after initial model download. No telemetry. Ever.
• Sensitive Data Filter — Auto-redacts credit cards, SSNs, API keys, passwords before storage.
• Encryption at Rest — AES encryption for screenshots (Fernet + OS keyring).
• Dashboard PIN Lock — Session-based auth with configurable auto-lock timeout.
• Incognito Mode — One-click pause. Nothing recorded.

⌨️ System-Wide Hotkeys

All hotkeys customizable from Settings.

🧠 How Gemma 4 Is Used

Gemma 4 E2B is not a bolt-on — it's architecturally load-bearing. ScreenMind uses all three modalities:

1. Vision — Screenshot Analysis

Every screenshot is sent to Gemma 4 with OCR context. It returns structured JSON:

• App name, activity category, summary, detailed context
• Mood classification, confidence score
• Rich scene description (every visible element inventoried)
• Layout regions (sidebar, chat area, toolbar boundaries)

Three modes (benchmarked on GTX 1650 4GB — scales dramatically with better GPUs):

• Accurate — single call with thinking (~76s). Best layout detection.
• Balanced — thinking enabled, analysis-only (~40s). Richer descriptions than Fast.
• Fast — no-thinking prefill trick (~12s). Layout via OCR clustering instead.

2. Audio — Voice Memos & Meeting Transcription

Gemma 4 E2B has a native audio encoder. ScreenMind uses it for:

• Voice memo transcription (hold hotkey → speak → release)
• Meeting transcription (15s chunks, map-reduce summarization for long meetings)

No Whisper dependency. One model handles everything.

3. Reasoning — Summaries, Chat, Agents

• Daily summaries with deep reasoning (think=True)
• Chat answers grounded in actual screen data (text-first RAG with vision fallback)
• Agent execution — Gemma processes markdown agent prompts with injected screen data

Why E2B Specifically?

Gemma 4 E2B is the only model that checks all five boxes.

🚀 Quick Start

Requirements: Python 3.10+ · GPU recommended (4GB+ VRAM) · ~5GB disk for model

1️⃣ Clone & Install

git clone https://github.com/ayushh0110/ScreenMind.git
cd ScreenMind

python -m venv venv
venv\Scripts\activate        # Windows
# source venv/bin/activate   # macOS/Linux

pip install -r requirements.txt

2️⃣ Run

3️⃣ Open → http://127.0.0.1:7777

On first run, ScreenMind will:

• Auto-detect your GPU and download llama-server if not found (CUDA/CPU auto-selected)
• Open the Model Hub — download Gemma 4 E2B GGUF (~5GB) with progress tracking right in the UI
• Chat and Summary stay locked (🧠💤 "I need my brain to think!") until the model is ready, then auto-unlock
• Start llama-server in background
• Show the welcome screen to set up an optional PIN
• Create ~/.screenmind/ for data storage

cp .env.example .env
# Edit capture interval, blocked apps, hotkeys, etc.

🏗️ Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                          ScreenMind                                  │
│                                                                     │
│  ┌────────────┐    ┌──────────────┐    ┌─────────────────────────┐ │
│  │  Capture   │───▶│  Async Queue │───▶│    Analysis Worker      │ │
│  │  Worker    │    │  (max: 100)  │    │                         │ │
│  │            │    └──────────────┘    │  ┌───────────────────┐  │ │
│  │ • Screen   │                        │  │  Per-App pHash    │  │ │
│  │ • Window   │                        │  │  Cache (3-tier)   │  │ │
│  │ • Dedup    │                        │  └───────────────────┘  │ │
│  │ • A11y     │                        │           │             │ │
│  │ • Privacy  │                        │           ▼             │ │
│  └────────────┘                        │  ┌───────────────────┐  │ │
│                                        │  │   EasyOCR         │  │ │
│  ┌────────────┐                        │  │   (text extract)  │  │ │
│  │   Audio    │                        │  └───────────────────┘  │ │
│  │   Worker   │                        │           │             │ │
│  │            │                        │           ▼             │ │
│  │ • Meeting  │                        │  ┌───────────────────┐  │ │
│  │   detect   │                        │  │   Gemma 4 E2B     │  │ │
│  │ • Record   │                        │  │   (via llama.cpp) │  │ │
│  │ • Transcr. │                        │  │   Vision + Audio  │  │ │
│  └────────────┘                        │  └───────────────────┘  │ │
│                                        │           │             │ │
│  ┌────────────┐                        │           ▼             │ │
│  │   Agent    │                        │  ┌───────────────────┐  │ │
│  │  Scheduler │                        │  │  Layout Analyzer  │  │ │
│  │            │                        │  │  (spatial OCR)    │  │ │
│  │ • .md AI   │                        │  └───────────────────┘  │ │
│  │ • .py code │                        │           │             │ │
│  └────────────┘                        │           ▼             │ │
│                                        │  ┌───────────────────┐  │ │
│                                        │  │  MiniLM-L6-v2     │  │ │
│                                        │  │  (embeddings)     │  │ │
│                                        │  └───────────────────┘  │ │
│                                        └─────────────────────────┘ │
│                                                    │               │
│                                                    ▼               │
│                                        ┌───────────────────┐       │
│                                        │   SQLite (WAL)    │       │
│                                        │   + FTS5 index    │       │
│                                        └─────────┬─────────┘       │
│                                                  │                 │
│  ┌───────────────────────────────────────────────┘                 │
│  │                                                                 │
│  ▼                                                                 │
│  ┌───────────────────────────────────────────────────────────────┐ │
│  │                    FastAPI REST Server                         │ │
│  │  /timeline · /search · /chat · /stats · /agents · /mcp       │ │
│  │                                                               │ │
│  │  ┌───────────────────────────────────────────────────────┐   │ │
│  │  │           Web Dashboard (Vanilla JS SPA)               │   │ │
│  │  │  Timeline · Chat · Search · Analytics · Agents · Settings │ │
│  │  └───────────────────────────────────────────────────────┘   │ │
│  └───────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘

Multi-Model AI Pipeline

Screenshot → EasyOCR (text) → Gemma 4 E2B (understanding) → MiniLM (embeddings) → SQLite + FTS5
                                     ↑
                              OCR text fed as context
                              (Gemma sees image + reads text)

Four AI models working in concert, with Gemma 4 as the brain:

1. EasyOCR — extracts raw screen text
2. Gemma 4 E2B — understands what you're doing (vision + reasoning)
3. MiniLM-L6-v2 — generates semantic vectors for natural language search
4. FTS5 — indexes text for instant keyword search

🤖 Agent Platform

ScreenMind includes a full agent/plugin system. Build any automation on top of your screen data.

Two Modes

Markdown Agent Example

---
name: Daily Focus Report
schedule: every 6h
data: timeline, apps, mood
output: local, obsidian
---

Analyze my screen activity and generate a focus report:
- How many hours of deep work vs shallow work?
- What were my main distractions?
- Give me a focus score out of 10.

Drop this file in ~/.screenmind/agents/ — it runs automatically.

Python Plugin SDK

from screenmind_sdk import ScreenMindSDK

sdk = ScreenMindSDK("my-tracker")

# Get today's activities filtered by app
activities = sdk.get_activities(app="Chrome", limit=20)

# Persistent state across runs
last_count = sdk.load_state("url_count", 0)
urls = sdk.get_urls_visited()
sdk.save_state("url_count", len(urls))

# Ask Gemma (GPU-safe — waits for idle)
insight = sdk.ask_gemma(f"Summarize these URLs: {urls}")
print(insight)

Data Selectors (Frontmatter)

Markdown agents declare what data they need:

Data injection auto-scales to your model's context window.

4 Agents Ship Built-In

• daily-journal.md — First-person journal entry from your day
• focus-report.md — Focus score, deep work hours, distractions
• meeting-actions.md — Extract action items from meeting transcripts
• code-changelog.md — Summarize coding activity (commits, files, repos)

🔌 MCP Server (Claude / Cursor / VS Code)

ScreenMind exposes your screen history to any MCP-compatible AI tool:

python mcp_server.py  # stdio transport

Claude Desktop config (~/.claude/claude_desktop_config.json):

{
  "mcpServers": {
    "screenmind": {
      "command": "python",
      "args": ["C:/path/to/screenmind/mcp_server.py"]
    }
  }
}

Tools Available

📡 API Reference

Full Swagger docs at http://127.0.0.1:7777/docs

Key Endpoints

🔧 Tech Stack

screenmind/
├── main.py                    # Entry point — starts all services
├── config.py                  # Pydantic settings (env + runtime overrides)
├── setup_llama.py             # Auto-detect + install llama-server
├── requirements.txt           # Full Python dependencies
├── requirements-test.txt      # Lightweight CI deps (no PyTorch)
├── mcp_server.py              # MCP server for Claude/Cursor/VS Code
├── screenmind_sdk.py          # SDK for Python plugin agents
│
├── capture/                   # Screenshot capture layer
│   ├── screen.py              # mss-based capture + encryption
│   ├── window.py              # Active window detection
│   ├── dedup.py               # Perceptual hash deduplication
│   ├── hotkey.py              # Global hotkeys (bookmark, pause, voice)
│   └── voice_recorder.py      # Mic recording for voice memos
│
├── engine/                    # AI & intelligence layer
│   ├── analyzer.py            # Gemma 4 vision analysis (dual mode)
│   ├── llm_client.py          # llama-server client (chat, vision, audio)
│   ├── model_manager.py       # Server lifecycle, model download/switch
│   ├── embedder.py            # MiniLM semantic embeddings
│   ├── ocr.py                 # EasyOCR text extraction
│   ├── layout_analyzer.py     # Spatial OCR organization
│   ├── dev_context.py         # Git repo/branch/diff detection
│   ├── a11y_extractor.py      # Accessibility API text extraction
│   └── agent_runner.py        # Agent scheduling & execution
│
├── workers/                   # Background processing
│   ├── capture_worker.py      # Smart capture loop + privacy filtering
│   ├── analysis_worker.py     # OCR → Gemma → Layout → Embed → Store
│   └── audio_worker.py        # Meeting detection & transcription
│
├── storage/                   # Data persistence
│   ├── database.py            # SQLite + FTS5 + migrations
│   └── models.py              # Pydantic data models
│
├── privacy/                   # Privacy & security
│   ├── encryption.py          # Fernet AES encryption at rest
│   └── data_filter.py         # Sensitive data redaction
│
├── platform_support/          # Cross-platform abstraction
│   ├── windows.py             # Win32 + UI Automation
│   ├── macos.py               # AppKit + AXUIElement
│   └── linux.py               # xdotool + AT-SPI
│
├── integrations/              # External connections
│   ├── obsidian.py            # Vault markdown export
│   ├── notion.py              # Notion API export
│   ├── webhooks.py            # HTTP webhooks (HMAC, retry)
│   └── smart_notify.py        # Distraction/break notifications
│
├── api/                       # REST API + dashboard
│   ├── server.py              # FastAPI app + auth middleware
│   ├── dependencies.py        # Shared state for routes
│   ├── routes/                # 16 route modules
│   └── static/                # Web dashboard (HTML + CSS + JS)
│
├── default_agents/            # 4 built-in agents
│   ├── daily-journal.md
│   ├── focus-report.md
│   ├── meeting-actions.md
│   └── code-changelog.md
│
├── tests/                     # pytest test suite (25 modules)
│   ├── conftest.py            # Shared fixtures
│   └── test_*.py              # Unit + integration tests
│
└── docs/
    └── BUILD_YOUR_OWN_AGENT.md

🛡️ Error Handling & Resilience

🎨 Dashboard

The web dashboard at http://127.0.0.1:7777 features:

• Timeline — Browse activities by date with thumbnails, AI summaries, category badges
• Chat — Conversational AI with screen memory. Ask anything about your history. Locked with 🧠💤 brain animation until model is ready.
• Search — Semantic + keyword hybrid search with OCR highlighting on screenshots
• Analytics — Category charts, top apps, hourly heatmap, meeting stats
• Rewind — Timelapse player with play/pause/scrub/speed controls
• Memos — Voice memo list with audio player
• Agents — Create, edit, run, and monitor agents
• Settings — Model Hub (download/switch models with progress), Shortcuts, Capture, AI, Audio, Privacy, Automation, Integrations, Storage

Dark glassmorphism UI. No build step. Instant load.

🧪 Development

Run the test suite:

# Fast (lightweight deps — same as CI, ~2 min install)
pip install -r requirements-test.txt
pytest --cov=. --cov-report=term-missing -q

# Full (includes ML models — sentence-transformers, easyocr)
pip install -r requirements.txt
pip install pytest pytest-asyncio pytest-cov
pytest --cov=. --cov-report=term-missing -q

CI runs automatically on push/PR via GitHub Actions using the lightweight deps.

🤝 Contributing

Contributions welcome! Here are some high-impact areas:

• 🍎 macOS/Linux testing — platform adapters exist, need real hardware testing
• 🐳 Docker container — one-command setup
• 🧩 Community agent registry — share agents between users
• 🌐 Browser extension — richer URL/tab context
• 📤 Export formats — Markdown, CSV, JSON

⭐ Show Your Support

If you find ScreenMind useful, please consider:

• ⭐ Star this repo — it helps others discover the project
• 🍴 Fork it — build your own agents and features
• 🐛 Report issues — help us improve
• 📣 Share it — tell others about privacy-first AI

📝 License

MIT License — see LICENSE for details.