VideoRAE : Taming Video Foundation Models for Generative Modeling via Representation Autoencoders

Zhihao Xie1,2,*, Junfeng Wu2,*, Xinting Hu4, Junchao Huang1,3, Li Jiang1,3,†
1The Chinese University of Hong Kong, Shenzhen, 2Huazhong University of Science and Technology, 3Shenzhen Loop Area Institute, 4University of Science and Technology of China
*Equal contribution. †Corresponding author.
Code
Overall architecture of VideoRAE.

Overall architecture of VideoRAE. A frozen VFM extracts multi-scale features, a 1D projector compresses them into compact latents (continuous or discrete), and the decoder is trained with reconstruction and REPA alignment.

Abstract

The rapid advancement of video generative modeling has been largely driven by diffusion and autoregressive models operating in the latent spaces of 3D Variational Autoencoders (3D-VAEs). However, conventional 3D-VAEs are mainly optimized for pixel-level reconstruction, and this objective often limits the semantic and spatio-temporal structure captured by their latent spaces, constraining downstream synthesis quality. Concurrently, Video Foundation Models (VFMs) such as V-JEPA 2 and VideoMAEv2 have demonstrated exceptional semantic understanding capabilities. Yet, whether such frozen video foundation representations can be transformed into compact, reconstruction-capable, and generation-friendly video latents remains largely unexplored. In this work, we answer this question affirmatively with VideoRAE. VideoRAE leverages multi-scale hierarchical features from a frozen video foundation encoder and employs a lightweight 1D self-attention projector to compress them into a highly compact latent space. The resulting latents support both continuous representations for Diffusion Transformers (DiTs) and discrete representations for autoregressive models via multi-codebook high-dimensional quantization. During decoding, VideoRAE incorporates a local-and-global representation alignment objective with the frozen VFM teacher, which improves semantic preservation and enables training without KL regularization. Comprehensive experiments show that VideoRAE achieves strong reconstruction in both continuous and discrete regimes. On UCF-101, it achieves state-of-the-art class-to-video gFVDs of 40 and 93 with AR and DiT generators, respectively, while converging approximately 5× faster than competing autoencoder baselines. In a controlled 2B-scale text-to-video experiment, replacing LTX-VAE with VideoRAE leads to faster convergence and consistently better VBench performance. These results validate frozen VFM representations as versatile and generation-friendly video latents. The model and code will be released.

Training Convergence

Training convergence speed comparison in AR (left) and DiT (right) generative modeling. VideoRAE converges approximately 5× faster than competing autoencoder baselines.

AR training convergence comparison.

AR

DiT training convergence comparison.

DiT

VBench Convergence

Convergence comparison between VideoRAE and LTX-VAE on VBench.

VBench semantic score convergence.

Semantic Score

VBench quality score convergence.

Quality Score

VBench total score convergence.

Total Score