In a previous article, we benchmarked three open-source Vision-Language Models on zero-shot object detection and arrived at an uncomfortable conclusion: even the fastest contender, Phi-3.5-vision-instruct, takes 4.45 seconds per frame on an NVIDIA L4. LLaVA-v1.6 sits at 8.13 seconds. For any application that needs to process a live video stream, these numbers are disqualifying. But the conclusion that VLMs are fundamentally incompatible with real-time workloads deserves more scrutiny. That 8-second figure was measured on a general-purpose zero-shot detection task, asking the model to reason about arbitrary objects in arbitrary scenes. What happens when you constrain the problem? When you give the model a closed vocabulary, a fixed resolution, a deterministic decoding strategy, and a non-blocking inference pipeline?
This article answers that question. Using PaliGemma, Google’s compact vision-language model, we built a real-time video classification system running at approximately 0.8 to 1.2 seconds per frame on an NVIDIA RTX A4500. That is a six to eight times improvement over LLaVA on comparable professional hardware, achieved entirely through architectural decisions rather than hardware upgrades. Here are the four patterns that made it possible.

Why PaliGemma

Before getting into architecture, the choice of model itself deserves an explanation, because PaliGemma is significantly underrepresented in the developer literature relative to its practical value. PaliGemma is a 3-billion parameter vision-language model built by Google, combining a SigLIP vision encoder with a Gemma language backbone. Compared to LLaVA-7B or Phi-3.5-vision, it is roughly half the size, which translates directly to lower VRAM consumption and faster inference on the same hardware. More importantly for classification tasks, it was explicitly fine-tuned on a wide range of visual understanding benchmarks including image captioning, visual question answering, and object localization, which means it has strong priors for the kind of constrained, structured responses we are going to elicit.

import torch
from transformers import PaliGemmaProcessor, PaliGemmaForConditionalGeneration

MODEL_PATH = "./paligemma_offline"

model = PaliGemmaForConditionalGeneration.from_pretrained(
    MODEL_PATH,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    local_files_only=True
).eval()

processor = PaliGemmaProcessor.from_pretrained(
    MODEL_PATH,
    local_files_only=True
)

Two details here are worth noting. Loading in bfloat16 rather than float32 halves the VRAM footprint with negligible accuracy degradation on classification tasks. The local_files_only=True flag is not just a convenience for offline environments: in a production system, it eliminates the network round-trip on initialization and guarantees that your inference environment is fully reproducible.

Pattern 1: resolution as a latency knob

The single most impactful decision in a real-time VLM pipeline is input resolution. VLMs process images by dividing them into patches and encoding each patch as a sequence of tokens. A 1280×720 frame generates a dramatically larger token sequence than a 448×448 crop, and since transformer attention scales quadratically with sequence length, resolution is not a linear cost: it is an exponential one. For zero-shot object detection, where spatial precision matters, downsampling is a real trade-off. But for scene-level classification tasks, where you are asking “what is the dominant emotion in this frame?” rather than “give me the pixel coordinates of every object,” 448×448 preserves more than enough semantic information.

import cv2
import numpy as np
from PIL import Image

def preprocess_frame(frame_bgr: np.ndarray) -> Image.Image:
    # Resize to inference resolution before any VLM processing
    frame_small = cv2.resize(frame_bgr, (448, 448))
    # Convert BGR (OpenCV) to RGB (PIL/transformers)
    return Image.fromarray(cv2.cvtColor(frame_small, cv2.COLOR_BGR2RGB))

The key insight is that resolution should be chosen based on the granularity of information your task actually requires, not based on the resolution of your input stream. If your camera captures at 1080p but your classification task only needs to distinguish between five emotional states, you are paying a massive compute tax for information you will never use.

Pattern 2: deterministic decoding with a closed vocabulary

Standard VLM usage treats the model as an open-ended text generator. You prompt it, it samples from a probability distribution, and you receive a natural language response that you then parse. This is the source of the fragility problem we discussed in the previous article, and it is also a significant source of latency: sampling with high max_new_tokens means the model runs the full autoregressive loop for every token it generates. For classification tasks, you can break this entirely. Instead of asking the model to describe what it sees, you constrain its output to a fixed vocabulary of valid labels and limit generation to the minimum number of tokens needed to express one of them.

# A generic set of states tailored to your specific domain
VALID_CLASSES = ['active', 'idle', 'error', 'offline', 'unknown']

def classify_frame(model: torch.nn.Module, processor, image: Image.Image) -> str:
    prompt = (
        f"<image> What is the current operational state shown in this frame? "
        f"You MUST choose ONLY ONE from: {VALID_CLASSES}."
    )

    inputs = processor(
        text=prompt,
        images=image,
        return_tensors="pt"
    ).to(model.device, model.dtype)

    with torch.inference_mode():
        output_ids = model.generate(
            **inputs,
            max_new_tokens=10,   # A single class label needs at most 2-3 tokens
            do_sample=False      # Greedy decoding: deterministic, faster, no temperature needed
        )

    raw_output = processor.decode(
        output_ids[0][inputs.input_ids.shape[-1]:],
        skip_special_tokens=True
    ).strip().lower()

    # Sanitize to alphabetic characters only
    return ''.join(filter(str.isalpha, raw_output))

Setting do_sample=False switches the model to greedy decoding, which always selects the highest-probability token at each step. This eliminates the sampling overhead and makes the output fully deterministic: identical inputs will always produce identical outputs, which is essential for debugging and for the temporal smoothing pattern we cover next. The max_new_tokens=10 cap means the model stops generating almost immediately after producing the label, rather than continuing to produce explanatory text nobody asked for.
The result is that you are using a 3B-parameter VLM as a highly capable, semantically-aware classifier rather than a generative model. You get the zero-shot flexibility of natural language prompting with inference characteristics that approach those of a dedicated classification head.

Pattern 3: temporal smoothing for prediction stability

Even with deterministic decoding, a VLM processing live video will produce noisy predictions. Lighting changes, motion blur, partial occlusion, and transient visual artifacts will cause the model to output inconsistent labels across consecutive frames. If you pipe raw per-frame predictions directly to a downstream system, you get jittery, unreliable output. The solution is temporal smoothing: instead of trusting any single prediction, you maintain a rolling window of recent predictions and emit the majority vote.

from collections import deque, Counter

class TemporalSmoother:
    def __init__(self, window_size: int = 5):
        self.history = deque(maxlen=window_size)

    def update(self, prediction: str) -> str:
        self.history.append(prediction)
        # Return the most common prediction in the window
        return Counter(self.history).most_common(1)[0][0]

A window of 5 frames at our inference rate translates to roughly 4 to 6 seconds of temporal context. This is enough to absorb transient noise while remaining responsive to genuine state changes. The window size is the primary tuning parameter: larger windows are more stable but slower to react; smaller windows are more responsive but noisier. For most classification tasks, 3 to 7 frames covers the practical range.

Pattern 4: non-blocking inference with a decoupled shared state architecture

The previous three patterns optimize the inference call itself. This one addresses a more fundamental systemic issue: a VLM inference call that takes 0.8 to 1.2 seconds will block any thread it runs on for that entire duration. If your video capture and your inference run on the same thread, your stream will stutter at the inference rate rather than the camera rate.

The naive solution is to use a standard Python queue.Queue to pass frames between threads. However, this introduces consumer-competition: if a rendering thread and an AI thread both read from the same queue, they consume the frames, stealing data from one another and causing severe visual stuttering and skipped inference cycles. The production-grade solution is an Asynchronous Shared State Pattern with granular locking. The video capture thread acts as a producer, continuously overwriting a shared “latest frame” pointer. The rendering thread (running on the main thread, which is mandatory for OpenCV UI operations on macOS and Wayland) and the AI background thread act as independent consumers, copying the latest frame into local memory whenever they are ready for their next cycle.

import threading
import time
import numpy as np
import cv2
import torch
from typing import Optional, Any

class SharedState:
    """
    Thread-safe state container.
    The lock is strictly granular: it is only held for memory assignment/copying,
    never during expensive I/O or AI inference operations.
    """
    def __init__(self):
        self.latest_frame: Optional[np.ndarray] = None
        self.prediction: str = "WAITING"
        self.lock = threading.Lock()
        self.running: bool = True

shared = SharedState()

def video_capture_worker(source: int = 0) -> None:
    """Reads frames at hardware speed and updates the shared state."""
    cap = cv2.VideoCapture(source)

    while shared.running:
        ret, frame = cap.read()
        if not ret:
            time.sleep(0.01)
            continue

        with shared.lock:
            # Overwrite with the freshest data.
            # Pointer assignment is fast enough to barely hold the lock.
            shared.latest_frame = frame

    cap.release()

def inference_worker(model: torch.nn.Module, processor: Any) -> None:
    """Consumes the latest frame at the AI's maximum throughput rate."""
    smoother = TemporalSmoother(window_size=5)

    while shared.running:
        with shared.lock:
            # Deep copy to prevent OpenCV from mutating the array during inference
            frame = shared.latest_frame.copy() if shared.latest_frame is not None else None

        if frame is None:
            time.sleep(0.05)
            continue

        try:
            image = preprocess_frame(frame)
            raw_pred = classify_frame(model, processor, image)
            smoothed_pred = smoother.update(raw_pred)

            with shared.lock:
                shared.prediction = smoothed_pred

        except torch.cuda.OutOfMemoryError:
            # Handle temporary VRAM spikes gracefully without killing the thread
            with shared.lock:
                shared.prediction = "OOM_ERROR"
            time.sleep(1.0)

        except Exception as e:
            # Catch corrupt frames or tensor mismatches
            with shared.lock:
                shared.prediction = "ERROR"

# Initialize background workers as Daemon threads
threads = [
    threading.Thread(target=video_capture_worker, args=(0,), daemon=True),
    threading.Thread(target=inference_worker, args=(model, processor), daemon=True),
]

for t in threads:
    t.start()

# Main Thread UI Loop
# UI libraries (cv2.imshow) must run on the main thread to prevent OS-level crashes.
while shared.running:
    with shared.lock:
        frame = shared.latest_frame.copy() if shared.latest_frame is not None else None
        label = shared.prediction

    if frame is not None:
        cv2.putText(frame, label.upper(), (30, 50),
                    cv2.FONT_HERSHEY_SIMPLEX, 1.5, (0, 255, 0), 3)
        cv2.imshow("Live Classification", frame)

    # 30 FPS rendering limit (33ms) + graceful shutdown
    if cv2.waitKey(33) & 0xFF == ord('q'):
        shared.running = False

cv2.destroyAllWindows()

The critical design principle here is granular locking: the lock is acquired, the numpy array is copied in memory (which takes microseconds), and the lock is instantly released. Holding the lock across a one-second VLM inference call would serialize all three components and defeat the entire purpose of the architecture. With this structure, your video capture thread runs at hardware framerate (e.g., 30 fps), your rendering loop displays frames at 30 fps, and your inference thread runs at its own async rate (1 fps). The three systems are temporally independent, limited only by their respective hardware bounds.

Benchmark summary

Running the complete pipeline on an NVIDIA RTX A4500 (20GB GDDR6, Ampere architecture) with PaliGemma in bfloat16 across a three-stream live video scenario yields a highly stable performance profile. By restricting the input resolution to 448 × 448 and capping the output at a maximum of 10 new tokens via a greedy decoding strategy (do_sample=False), the system achieves an inference latency between 0.8 and 1.2 seconds per frame. Combined with a 5-frame temporal smoothing window, this configuration ensures reliable state classification while the decoupled architecture allows the video capture thread to maintain a steady 25 fps, completely independent of the inference bottleneck.

For comparison, LLaVA-v1.6-Mistral-7B running open-vocabulary zero-shot detection on an NVIDIA L4 clocks in at 8.13 seconds per frame. While the hardware is not directly equivalent, the magnitude of the gap confirms that architectural constraints, rather than raw compute, account for the vast majority of the difference.

When this architecture makes sense

This pattern is a strong fit when your task is classifiable into a fixed label set, you need continuous processing of a live stream rather than batch analysis of static images, data privacy requirements preclude sending frames to an external API, and you can tolerate sub-second rather than sub-100ms latency. It is not the right tool when you need genuine real-time response at conveyor-belt speeds, where sub-50ms latency is non-negotiable. In that regime, you are back in YOLO territory, or you use a pipeline like the one described in the previous article: leverage the VLM to auto-annotate a dataset overnight, then train a dedicated lightweight classifier for production deployment.

Conclusion

The gap between “VLMs are too slow for video” and “VLMs work in production video pipelines” is not primarily a hardware problem. It is an architectural one. Choosing a compact model like PaliGemma over a 7B-parameter alternative, constraining resolution to what your task actually requires, enforcing deterministic decoding over a closed vocabulary, smoothing predictions temporally, and decoupling inference from capture and rendering: none of these require a bigger GPU. They require thinking carefully about what you are actually asking the model to do, and building your pipeline around that constraint rather than against it.

The full pattern, from model loading to multi-threaded inference, fits in under 150 lines of Python. That is a reasonable price for zero-shot semantic classification on a live video stream.