Cong Wei, PhD candidate
David R. Cheriton School of Computer Science
Supervisor: Professor Wenhu Chen
Unified multimodal models have shown promising capabilities in understanding, generating, and editing visual content. However, most existing approaches remain limited to the image domain or rely on separate architectures and representations for understanding and generation.
In this seminar, I will first introduce UniVideo, a unified framework that extends omni modeling to the video domain. UniVideo combines a Multimodal Large Language Model for interpreting complex text and visual instructions with a Multimodal Diffusion Transformer for video synthesis. Under a unified instruction paradigm, it supports text-to-video and image-to-video generation, in-context video generation, and video editing. Its unified design also enables capability composition and generalization to unseen editing instructions.
I will then introduce OmniPixel, a fully pixel-space omni model that unifies text-to-image generation, text-to-video generation, image understanding, and video understanding within a single framework. Unlike prior unified models that depend on pretrained vision encoders such as ViTs for understanding and VAEs for visual generation, OmniPixel removes both components and learns directly from raw pixels. This enables end-to-end optimization of visual understanding and generation in a raw pixel-space model.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。