Introduction
"Video is the last blue ocean of data and the most challenging source of unstructured information."
This is the No.66 article in the "One Open Source Project a Day" series. Today we are exploring NVIDIA Video Search and Summarization (VSS).
In traditional vision surveillance or video analytics, we usually rely on specific object detection algorithms (e.g., "detecting people and cars"). However, when we need to find "a person wearing a red shirt, holding a blue coffee cup, and walking towards the meeting room," traditional rule-driven systems often fail. NVIDIA VSS provides a comprehensive reference architecture that integrates Vision Language Models (VLMs) and Large Language Models (LLMs), allowing developers to build Vision Agents that "understand" video content like a human.
What You Will Learn
- Multimodal Workflow: How to perform search and semantic analysis on video using natural language.
- NVIDIA NIM Microservices: Leveraging high-performance inference containers to accelerate vision tasks.
- RTVI Architecture: Understanding the indexing and processing flow of Real-Time Video Intelligence.
- MCP Integration: How to use the Model Context Protocol to manage video analytics tools uniformly.
- Enterprise Deployment: Rapid implementation from cloud to local GPU clusters.
Prerequisites
- Basic understanding of LLMs and Vision Language Models (VLMs).
- Familiarity with Docker and basic hardware operations (especially NVIDIA GPUs).
- Knowledge of what Vector Databases do in RAG (Retrieval-Augmented Generation) systems.
Project Background
Project Introduction
NVIDIA Video Search and Summarization (VSS) is a core project within the NVIDIA AI Blueprints suite. It is not just a library, but an enterprise-grade reference architecture. It addresses the pain point of converting raw audio-video streams into structured, searchable insights, enabling users to "talk" to their video data through a chat interface to search for specific moments, generate summaries, or perform visual Q&A.
Author/Team Introduction
- Author: NVIDIA Metropolis / AI Blueprints Team
- Background: NVIDIA is the world leader in AI computing. The Metropolis team focuses on vision AI solutions for smart cities, industrial automation, and retail insights.
- Release Date: 2024-2025 (latest VSS 3.1.0 updated in March 2026).
Project Data
- ⭐ GitHub Stars: 1.2k+
- 🍴 Forks: 260+
- 📄 License: NVIDIA AI Product Agreement
- 📦 Version: v3.1.0
- 🌐 Website: NVIDIA AI Blueprints
Main Features
Core Value
The core of VSS lies in "semanticizing" video content. It uses video encoders to extract features and store them in a vector index, combined with powerful VLMs (like Cosmos-Reason2-8B) to achieve deep understanding across video streams.
Use Cases
- Smart Retail & Spaces: Analyzing customer paths or identifying on-site safety hazards.
- Warehouse & Industrial Automation: Validating Standard Operating Procedures (SOPs) via video.
- Security Surveillance Synergy: Visually verifying real-time alerts and filtering out false positives from traditional algorithms using natural language.
- Digital Asset Management: Quickly locating specific shots in massive historical video archives and exporting summary reports.
Quick Start
You will need a machine with an NVIDIA GPU (RTX 6000 Ada or A100/H100 recommended) and an NVIDIA API Key.
# 1. Clone the repository
git clone https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization.git
cd video-search-and-summarization
# 2. Configure environment variables
echo "NVIDIA_API_KEY=your_key_here" > .env
# 3. Start the full stack using Docker Compose (UI, API, and Indexing engine)
docker compose up -d
Once started, access the UI at http://localhost:3000 (driven by Next.js) to upload videos or connect RTSP streams.
Core Features
- Natural Language Semantic Search: Supports complex queries like "find all people holding umbrellas in the rain."
- Visual Q&A: Ask detailed questions about specific clips, such as "Is the worker wearing a safety helmet?"
- Automated Video Summarization: Generates concise text summaries and keyframe lists for hours of footage.
- Real-Time Video Intelligence (RTVI): Supports low-latency embedding extraction from live streams.
- Tool Calling (MCP): The agent can dynamically call specialized analysis tools (e.g., counters, rangefinders) based on the context.
Project Advantages
| Feature | NVIDIA VSS | Open Source VLM Demos (e.g., LLaVA) | Traditional VMS |
|---|---|---|---|
| Pipeline Completeness | Full-stack (Index, Retrieval, UI) | Inference only, no video engineering | Basic rule filtering only |
| Real-time Performance | Optimized GPU pipeline (RTSP) | Mostly single file, high latency | Millisecond-level but lacks semantic |
| Scalability | Supports hundreds of concurrent streams | Resource intensive, hard to scale | Simple but functionally rigid |
Detailed Analysis
Architecture: RTVI + NIM
The architecture of VSS is known as RTVI (Real-Time Video Intelligence). It divides the processing into two planes:
1. Indexing Plane
Uses dedicated Vision Encoders (high-efficiency models built by NVIDIA) to convert every frame or second of video into vectors. These vectors, along with metadata, are stored in a high-performance vector index. This turns video "searching" into a large-scale vector retrieval task.
2. Inference Plane
When a user asks a question, the LLM acts as a controller, first fetching relevant video segments from the Indexing Plane, and then feeding those segments into a high-performance VLM (running on NVIDIA NIM microservices) for deep reasoning.
Key Components: Cosmos & Nemotron
- Cosmos-Reason2-8B: The core VLM responsible for understanding complex visual scenes and logical relationships.
- Nemotron-Nano-9B: A lightweight controller responsible for parsing natural language intent and converting it into tool calls.
MCP (Model Context Protocol)
VSS recently introduced MCP technology, allowing vision agents to seamlessly access external tools. For example, if a question involves "Is this car speeding?", the agent can dynamically call a professional speed-analysis plugin via the MCP interface, rather than just "estimating" based on visuals.
Links & Resources
Official Resources
- 🌟 GitHub: NVIDIA-AI-Blueprints/video-search-and-summarization
- 📚 Documentation: NVIDIA Metropolis Documentation
- 💬 Solution Guide: AI Blueprint for VSS
Target Audience
- Enterprise Developers: Building smart cities, industrial AI, or high-end surveillance systems.
- AI Engineers: Looking to learn how to implement VLMs in real-world video processing pipelines.
- Video Analysts: Users seeking automated, natural language interactive video reporting tools.
Visit my homepage to find more useful knowledge and interesting products.