512²×96 showcase. Input, reconstruction, and the tokens that survived across multiple clips. Black tiles are latent tokens dropped by the adaptive selector.
Contributions
Adaptive tokenization. A transformer video VAE learns a per-token keep-or-drop decision jointly with the latents, so each clip's token count is decided by its content — no preset budget, no inference-time search. The mask gates decoding as a soft attention mask, and an asymmetric coarse-to-fine decoder recovers detail from the compact latent set. On Panda-70M at 256²×16 this comes to 366 active tokens on average at 31.24 PSNR / 5.12 rFVD.
Sparse-token generation. Dropping tokens breaks spatial layout for downstream diffusion; a lightweight cascaded mask prior (and a joint content–position variant) restores it, reaching 61.53 gFVD on UCF-101 with 6.9× faster generator training than a dense transformer tokenizer.
Problem
Fixed token grids waste capacity on video redundancy.
A static background, a repeated frame, and a fast-moving object do not carry the same amount of information. Yet fixed-ratio video VAEs encode them with the same latent grid, so token count keeps scaling with space and time even when many regions are redundant.
Fixed grids keep every token; KATok keeps more where motion and detail need it.
Fixed tokenization
Tokens scale with H × W × T
The latent grid grows with resolution and duration, regardless of how much new information each patch contains.
Flexible is not adaptive
A budget still has to be chosen
Variable-length tokenizers can expose a token count, but selecting the right count per sample often needs search or extra overhead.
What KATok targets
Let content decide Neff
Adaptive tokenization should remove redundant tokens automatically while preserving enough structure for reconstruction and generation.
Method
A differentiable keep-or-drop bottleneck.
KATok leaves the VAE pipeline intact, but adds one learned decision at the latent bottleneck: for each encoded token, should it be kept or dropped?
The same mask gates latent values and decoder attention, while the finer decoder query grid reconstructs detail from compact coarse latents.
The selector predicts the latent posterior and keep/drop logits from the same encoder embedding, so sparsity is learned with the representation rather than applied after encoding.
KATok also uses asymmetric patching: the encoder tokenizes videos coarsely with 16²×8 patches, while the decoder reconstructs on a finer 8²×4 query grid. The latent set stays compact, but reconstruction can still recover finer detail.
Training. Gumbel-Softmax keeps the token mask differentiable, and the decoder receives it as a soft attention mask while a sparsity loss rewards compact token sets. Inference. Hard masks remove redundant tokens before decoding, with the corresponding attention mask blocking dropped tokens in decoder attention.
Reconstruction
The mask concentrates capacity where the video changes.
This example is a 256²×16 clip encoded with 370 tokens in total: 368 content tokens plus 2 register tokens, reaching 30.05 PSNR. Black tiles are dropped latent patches. KATok retains tokens around motion and detailed regions, while static or homogeneous areas are aggressively compressed.
Input
Reconstruction
Kept tokens
Results
Compact representation without giving up fidelity.
On Panda-70M validation, KATok improves reconstruction metrics while using far fewer active tokens than fixed or search-based baselines in the same continuous-token VAE family.
At 256²×16, KATok encodes each video with 366 active tokens on average.
Method
Resolution
#Tokens
Channels
Comp. ↑
PSNR ↑
LPIPS ↓
SSIM ↑
rFVD ↓
OmniTokenizer-VAE
256²×17
5120
8
96
28.10
0.05
0.88
7.84
ElasticTok-KL
256²×16
3845.56
8
102.25
30.52
0.06
0.91
12.37
KATok
256²×16
366.24
64
134.21
31.24
0.04
0.94
5.12
OmniTokenizer-VAE
512²×33
36864
8
96
24.07
0.06
0.80
16.85
KATok
512²×32
1554.24
64
253.00
33.23
0.05
0.95
6.40
Panda-70M validation. Comp. is H·W·T·3 / (#tokens · channels); #Tokens is the average active-token count for adaptive methods.
Generation
Sparse latents need spatial grounding.
KATok uses the learned VAE as the tokenizer for downstream flow-matching video generation. After token dropping, the generator must recover not only what each sparse token contains, but also where it belongs.
Why positions matter
Adaptive tokenization removes redundant grid cells, so sparse latent tokens no longer carry an implicit dense layout. For generation, KATok must model both the latent content and the positions where those latents should be decoded.
Position-aware sparse generation
KATok studies two variants: a cascaded mask prior that predicts active positions before content generation, and a joint variant that denoises content and coordinates together.
Cascaded mask-prior conditioning. The default strategy first predicts a binary occupancy mask, extracts active positions, and conditions the sparse content generator on those positions.Joint content-position generation. The ablation concatenates 3D coordinates with latent content and denoises them together with decoupled noise schedules.
Content-position misalignment
A content-only flow model can produce plausible sparse token values while placing them at the wrong spatio-temporal locations, leading to unstable boundaries and temporal inconsistencies.
Fig. 9. Naive sparse generation can misalign content and position; joint and cascaded generation reduce the mismatch.
Using the same tokenizer and SiT-XL content model (686.29M parameters) on UCF-101, gFVD improves from 95.69 with naive sparse flow matching to 73.16 with joint content-position generation (686.59M total) and 61.53 with cascaded mask-prior conditioning (694.65M total, including an 8.3M mask prior).
Generation
Emergent token-count control.
Token count can also modulate generation: higher budgets produce richer motion and detail. Each grid shows 16 unconditional SkyTimelapse samples from the cascaded model — same settings, only the token budget differs.
200 tokens
400 tokens
First 16 samples of each budget, no curation. Low budgets yield calm, mostly clear skies; higher budgets bring denser cloud structure and stronger motion.
Citation
BibTeX
@inproceedings{lee2026katok,
title = {Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation},
author = {Lee, Yeonkyeong and Go, Hyunsung and Kim, Jongmin and Lim, Sewoong and Lee, Donghoon},
booktitle = {European Conference on Computer Vision (ECCV)},
year = {2026}
}
