VideoRAE

Abstract

Compress frozen VFM features into compact continuous and discrete latents for video generation.

Backbones

V-JEPA 2 V-JEPA 2.1 VideoMAEv2

Latent spaces

Discrete Continuous

Downstream

Text-to-Video Class-to-Video

The rapid advancement of video generative modeling has been largely driven by diffusion and autoregressive models operating in the latent spaces of 3D Variational Autoencoders (3D-VAEs). However, conventional 3D-VAEs are mainly optimized for pixel-level reconstruction, and this objective often limits the semantic and spatio-temporal structure captured by their latent spaces, constraining downstream synthesis quality. Concurrently, Video Foundation Models (VFMs) such as V-JEPA 2 and VideoMAEv2 have demonstrated exceptional semantic understanding capabilities. Yet, whether such frozen video foundation representations can be transformed into compact, reconstruction-capable, and generation-friendly video latents remains largely unexplored. In this work, we answer this question affirmatively with VideoRAE. VideoRAE leverages multi-scale hierarchical features from a frozen video foundation encoder and employs a lightweight 1D self-attention projector to compress them into a highly compact latent space. The resulting latents support both continuous representations for Diffusion Transformers (DiTs) and discrete representations for autoregressive models via multi-codebook high-dimensional quantization. During decoding, VideoRAE incorporates a local-and-global representation alignment objective with the frozen VFM teacher, which improves semantic preservation and enables training without KL regularization. Comprehensive experiments show that VideoRAE achieves strong reconstruction in both continuous and discrete regimes. On UCF-101, it achieves state-of-the-art class-to-video gFVDs of 40 and 93 with AR and DiT generators, respectively, while converging approximately 5× faster than competing autoencoder baselines. In a controlled 2B-scale text-to-video experiment, replacing LTX-VAE with VideoRAE leads to faster convergence and consistently better VBench performance. These results validate frozen VFM representations as versatile and generation-friendly video latents. The model and code will be released.

Method

How VideoRAE builds generation-ready latents.

01
Frozen VFM features

Extract multi-scale hierarchical features from a frozen video foundation encoder and fuse them.
02
Compact latents

Compress with a 1D self-attention projector into continuous DiT latents or discrete AR tokens.
03
Align & decode

Align decoder features with the VFM teacher at local and global scales to preserve semantics.

Given an input video, multi-scale hierarchical features are first extracted from a frozen VFM and fused, followed by a 1D self-attention projector that dynamically compresses them into compact base tokens. These tokens are then formatted into either a continuous latent space via linear projection for Diffusion Transformers, or a discrete latent space via Multi-Codebook SimVQ for Autoregressive models. Finally, during decoding, the Representation Alignment module explicitly aligns intermediate decoder features with the VFM teacher at both local and global scales, intrinsically regularizing the semantic structure of the latent manifold.

Evaluation

Experimental Results

40

AR gFVD on UCF-101

State-of-the-art discrete generation

93

DiT gFVD on UCF-101

State-of-the-art continuous generation

5×

Faster convergence

vs. competing autoencoder baselines

Reconstruction

VideoRAE delivers strong reconstruction in both continuous and discrete regimes, preserving fine spatial details and temporal consistency under a highly compact latent budget.

Table 1. Reconstruction performance of discrete tokenizers on UCF-101 and TokenBench.

VideoRAE (VideoMAEv2) VideoRAE (V-JEPA 2) VideoRAE (V-JEPA 2.1)

Method	#rToken	UCF-101			TokenBench-256×256
Method	#rToken	PSNR↑	LPIPS↓	rFVD↓	PSNR↑	LPIPS↓	rFVD↓
AR generative models with discrete video tokenizers
TATS	1024	—	—	162	—	—	—
OmniTokenizer	5120	29.25	0.11	28	25.55	0.11	44
LARP-L-Long	1024	27.88	0.12	35	28.65	0.11	45
SweetTok	1280	29.27	0.07	20	—	—	—
VideoRAE (VideoMAEv2)	1024	29.94	0.10	16	30.69	0.09	33
VideoRAE (V-JEPA 2)	1024	29.39	0.10	13	29.93	0.10	28

Table 2. Reconstruction performance of continuous autoencoders on UCF-101 and TokenBench.

Method	Config	UCF-101			TokenBench-256×256
Method	Config	PSNR↑	LPIPS↓	rFVD↓	PSNR↑	LPIPS↓	rFVD↓
Diffusion generative models with continuous video tokenizers
CV-VAE	4096×4	30.11	0.12	57	30.80	0.11	61
LTX-VAE	128×128	32.02	0.10	35	33.74	0.08	32
LTX2	128×128	28.88	0.12	15	29.36	0.11	36
LeanVAE	4096×4	30.99	0.11	10	31.79	0.10	23
VideoRAE (VideoMAEv2)	512×32	31.23	0.08	13	32.06	0.08	25
VideoRAE (V-JEPA 2)	512×32	30.40	0.09	14	31.19	0.09	25

CogVideoX-VAE	4096×16	36.22	0.05	7	36.35	0.04	9
WAN2.1-VAE	4096×16	35.70	0.05	4	37.02	0.04	6
VideoRAE (VideoMAEv2)	1024×64	33.64	0.06	5	34.25	0.07	11
VideoRAE (V-JEPA 2)	1024×64	32.14	0.07	7	32.56	0.08	13

Table 3. Reconstruction performance of 3D projector variants on UCF-101 and TokenBench.

Method	Config	UCF-101			TokenBench-256×256
Method	Config	PSNR↑	LPIPS↓	rFVD↓	PSNR↑	LPIPS↓	rFVD↓
VideoRAE-3d (VideoMAEv2)	1024×64	33.50	0.06	5	34.07	0.07	12
VideoRAE-3d (V-JEPA 2)	1024×64	31.91	0.07	7	32.48	0.08	15
VideoRAE-3d (V-JEPA 2.1)	1024×64	31.88	0.07	7	32.19	0.07	16

Generation

On UCF-101 class-to-video generation, VideoRAE reaches state-of-the-art gFVDs of 40 (AR) and 93 (DiT), outperforming competing autoencoder baselines under matched generator settings.

Table 4. Class-conditional generation evaluation (Discrete AR) on UCF-101, evaluated on 10K generated samples.

Tokenizer	Resolution	#Tokens	#Params	gFVD↓
CogVideo (Hong et al., 2022)	256×256	6800	9.4B	626
TATS (Ge et al., 2022)	256×256	4096	321M	332
Video-LaVIT (Jin et al., 2024)	256×256	512	7B	280
OmniTokenizer (Wang et al., 2024b)	256×256	5120	650M	191
LARP-L (Wang et al., 2024a)	256×256	1024	632M	99
SweetTok (Tan et al., 2025)	256×256	1280	650M	84
SweetTok (Tan et al., 2025)	256×256	1280	1.9B	65
VideoRAE (VideoMAEv2)	224×224	1024	1.3B	45
VideoRAE (V-JEPA 2)	256×256	1024	1.3B	40

Table 5. Class-conditional generation evaluation (Continuous DiT) on UCF-101, evaluated on 2K generated samples.

Method	Resolution	Configs	gFVD↓
Latte (Ma et al., 2024)	256×256	4096×4	478
LeanVAE (Cheng & Yuan, 2025)	256×256	4096×4	164
LTX-VAE (HaCohen et al., 2024)	256×256	128×128	161
VideoRAE (VideoMAEv2)	224×224	512×32	99
VideoRAE (V-JEPA 2)	256×256	512×32	93

WF-VAE (Li et al., 2025)	256×256	4096×16	371
LeanVAE (Cheng & Yuan, 2025)	256×256	4096×16	175
DeCo-VAE (Yin et al., 2025)	256×256	4096×16	166
CogVideoX-VAE (Yang et al., 2025)	256×256	4096×16	151
WAN2.1-VAE (Wan et al., 2025)	256×256	4096×16	126
VideoRAE (VideoMAEv2)	224×224	1024×64	103
VideoRAE (V-JEPA 2)	256×256	1024×64	94

Table 6. Class-conditional generation evaluation (3D projector variants) on UCF-101, evaluated on 2K generated samples.

Method	Resolution	Configs	gFVD↓
VideoRAE-3d (VideoMAEv2)	224×224	1024×64	104
VideoRAE-3d (V-JEPA 2)	256×256	1024×64	96
VideoRAE-3d (V-JEPA 2.1)	256×256	1024×64	93

Class-to-Video Convergence

VideoRAE converges approximately 5× faster than LTX-Video on class-to-video generation, matching baseline quality in far fewer training epochs for both AR and DiT settings.

Class-to-video AR convergence comparison. — AR

Class-to-video DiT convergence comparison. — DiT

Text-to-Video Convergence (VBench)

In a controlled 2B-scale text-to-video setting, replacing LTX-VAE with VideoRAE yields faster convergence and consistently higher VBench semantic, quality, and total scores.

VBench semantic score convergence. — Semantic Score

VBench quality score convergence. — Quality Score

VBench total score convergence. — Total Score

Visual showcase

Class-to-Video Generation

Visual showcase

Text-to-Video Generation

Citation

BibTeX

If you find this work useful, please consider citing VideoRAE.

BibTeX

@article{xie2026videorae,
  title={VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders},
  author={Xie, Zhihao and Wu, Junfeng and Hu, Xinting and Huang, Junchao and Jiang, Li},
  journal={arXiv preprint arXiv:2607.14088},
  year={2026}
}

VideoRAE Taming Video Foundation Models for Generative Modeling via Representation Autoencoders

Introducing VideoRAE

Compress frozen VFM features into compact continuous and discrete latents for video generation.

How VideoRAE builds generation-ready latents.

Frozen VFM features

Compact latents

Align & decode

Experimental Results

Reconstruction

Generation

Class-to-Video Convergence

Text-to-Video Convergence (VBench)

Class-to-Video Generation

Text-to-Video Generation

BibTeX