Introduction

"Video is the last blue ocean of data and the most challenging source of unstructured information."

This is the No.66 article in the "One Open Source Project a Day" series. Today we are exploring NVIDIA Video Search and Summarization (VSS).

In traditional vision surveillance or video analytics, we usually rely on specific object detection algorithms (e.g., "detecting people and cars"). However, when we need to find "a person wearing a red shirt, holding a blue coffee cup, and walking towards the meeting room," traditional rule-driven systems often fail. NVIDIA VSS provides a comprehensive reference architecture that integrates Vision Language Models (VLMs) and Large Language Models (LLMs), allowing developers to build Vision Agents that "understand" video content like a human.

What You Will Learn

• Multimodal Workflow: How to perform search and semantic analysis on video using natural language.
• NVIDIA NIM Microservices: Leveraging high-performance inference containers to accelerate vision tasks.
• RTVI Architecture: Understanding the indexing and processing flow of Real-Time Video Intelligence.
• MCP Integration: How to use the Model Context Protocol to manage video analytics tools uniformly.
• Enterprise Deployment: Rapid implementation from cloud to local GPU clusters.

Prerequisites

• Basic understanding of LLMs and Vision Language Models (VLMs).
• Familiarity with Docker and basic hardware operations (especially NVIDIA GPUs).
• Knowledge of what Vector Databases do in RAG (Retrieval-Augmented Generation) systems.

Project Background

Project Introduction

NVIDIA Video Search and Summarization (VSS) is a core project within the NVIDIA AI Blueprints suite. It is not just a library, but an enterprise-grade reference architecture. It addresses the pain point of converting raw audio-video streams into structured, searchable insights, enabling users to "talk" to their video data through a chat interface to search for specific moments, generate summaries, or perform visual Q&A.

Author/Team Introduction

• Author: NVIDIA Metropolis / AI Blueprints Team
• Background: NVIDIA is the world leader in AI computing. The Metropolis team focuses on vision AI solutions for smart cities, industrial automation, and retail insights.
• Release Date: 2024-2025 (latest VSS 3.1.0 updated in March 2026).

Project Data

• ⭐ GitHub Stars: 1.2k+
• 🍴 Forks: 260+
• 📄 License: NVIDIA AI Product Agreement
• 📦 Version: v3.1.0
• 🌐 Website: NVIDIA AI Blueprints

Main Features

Core Value

The core of VSS lies in "semanticizing" video content. It uses video encoders to extract features and store them in a vector index, combined with powerful VLMs (like Cosmos-Reason2-8B) to achieve deep understanding across video streams.

Use Cases

1. Smart Retail & Spaces: Analyzing customer paths or identifying on-site safety hazards.
2. Warehouse & Industrial Automation: Validating Standard Operating Procedures (SOPs) via video.
3. Security Surveillance Synergy: Visually verifying real-time alerts and filtering out false positives from traditional algorithms using natural language.
4. Digital Asset Management: Quickly locating specific shots in massive historical video archives and exporting summary reports.

Quick Start

You will need a machine with an NVIDIA GPU (RTX 6000 Ada or A100/H100 recommended) and an NVIDIA API Key.

# 1. Clone the repository
git clone https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization.git
cd video-search-and-summarization

# 2. Configure environment variables
echo "NVIDIA_API_KEY=your_key_here" > .env

# 3. Start the full stack using Docker Compose (UI, API, and Indexing engine)
docker compose up -d

Once started, access the UI at http://localhost:3000 (driven by Next.js) to upload videos or connect RTSP streams.

Core Features

1. Natural Language Semantic Search: Supports complex queries like "find all people holding umbrellas in the rain."
2. Visual Q&A: Ask detailed questions about specific clips, such as "Is the worker wearing a safety helmet?"
3. Automated Video Summarization: Generates concise text summaries and keyframe lists for hours of footage.
4. Real-Time Video Intelligence (RTVI): Supports low-latency embedding extraction from live streams.
5. Tool Calling (MCP): The agent can dynamically call specialized analysis tools (e.g., counters, rangefinders) based on the context.

Project Advantages

Detailed Analysis

Architecture: RTVI + NIM

The architecture of VSS is known as RTVI (Real-Time Video Intelligence). It divides the processing into two planes:

1. Indexing Plane

Uses dedicated Vision Encoders (high-efficiency models built by NVIDIA) to convert every frame or second of video into vectors. These vectors, along with metadata, are stored in a high-performance vector index. This turns video "searching" into a large-scale vector retrieval task.

2. Inference Plane

When a user asks a question, the LLM acts as a controller, first fetching relevant video segments from the Indexing Plane, and then feeding those segments into a high-performance VLM (running on NVIDIA NIM microservices) for deep reasoning.

Key Components: Cosmos & Nemotron

• Cosmos-Reason2-8B: The core VLM responsible for understanding complex visual scenes and logical relationships.
• Nemotron-Nano-9B: A lightweight controller responsible for parsing natural language intent and converting it into tool calls.

MCP (Model Context Protocol)

VSS recently introduced MCP technology, allowing vision agents to seamlessly access external tools. For example, if a question involves "Is this car speeding?", the agent can dynamically call a professional speed-analysis plugin via the MCP interface, rather than just "estimating" based on visuals.

Links & Resources

Official Resources

• 🌟 GitHub: NVIDIA-AI-Blueprints/video-search-and-summarization
• 📚 Documentation: NVIDIA Metropolis Documentation
• 💬 Solution Guide: AI Blueprint for VSS

Target Audience

• Enterprise Developers: Building smart cities, industrial AI, or high-end surveillance systems.
• AI Engineers: Looking to learn how to implement VLMs in real-world video processing pipelines.
• Video Analysts: Users seeking automated, natural language interactive video reporting tools.

Visit my homepage to find more useful knowledge and interesting products.