reAR: Rethinking Visual Autoregressive Models via Generator-Tokenizer Consistency Regularization

News

Abstract

Visual autoregressive (AR) generation offers a promising path toward unifying vision and language models, yet its performance remains suboptimal against diffusion models. Prior work often attributes this gap to tokenizer limitations and rasterization ordering. In this work, we identify a core bottleneck from the perspective of generator-tokenizer inconsistency, i.e., the AR-generated tokens may not be well-decoded by the tokenizer.

To address this, we propose reAR, a simple training strategy introducing a token-wise regularization objective: when predicting the next token, the causal transformer is also trained to recover the visual embedding of the current token and predict the embedding of the target token under a noisy context. It requires no changes to the tokenizer, generation order, inference pipeline, or external models.

Despite its simplicity, reAR substantially improves performance. On ImageNet, it reduces gFID from 3.02 to 1.86 and improves IS to 316.9 using a standard rasterization-based tokenizer. When applied to advanced tokenizers, it achieves a gFID of 1.42 with only 177M parameters, matching the performance with larger state-of-the-art diffusion models (675M).

Understanding the Bottleneck

Generator-Tokenizer Inconsistency

We identify two key sources of inconsistency between the autoregressive generator and the visual tokenizer:

1. Amplified Exposure Bias: During training with teacher forcing, the model predicts tokens given ground-truth context, but at inference it conditions on its own predictions. In visual AR, this leads to unseen token sequences that corrupt future predictions and spread structural artifacts across the image.

2. Embedding Unawareness: The AR model optimizes only discrete token indices without considering how these tokens are embedded by the tokenizer. However, decoded image quality depends on the embeddings of the generated tokens rather than their indices alone.

Amplified Exposure Bias

Embedding Unawareness

Token-wise Consistency Regularization

reAR addresses these issues through two complementary strategies: Noisy Context Regularization that exposes the model to perturbed context during training, and Codebook Embedding Regularization that aligns the generator's hidden states with the tokenizer's embedding space. This encourages the generator to be aware of how tokens are decoded into visual patches.

Results

Generation Quality

Table 1 shows that reAR achieves strong results even with a standard raster-order AR model and a simple 2D patch tokenizer. reAR-S outperforms prior raster AR models like LlamaGen-XL (FID 2.00 vs. 2.34; IS 295.7 vs. 253.9) using only 14% of the parameters (201M vs. 1.4B), and surpasses advanced-tokenizer AR models such as WeTok with just 13–15% of their size. It matches RAR and outperforms RandAR under similar scales, and reAR-L exceeds MAR-L and VAR-d30. While diffusion and masked-generation models remain strong, reAR narrows the gap with far fewer training epochs.

Table: Results on 256x256 class-conditional generation on ImageNet-1K. "Mask." indicates masked generation; "Tok." denotes non-standard tokenization; "Rand." denotes randomized order; "Raster." denotes rasterization order. "†" indicates that the model is not provided and it's trained with our implementation. BPP₁₆=16×BPP (bits per pixel) measures the compression rate of discrete tokenizers and is not applicable ("N/A") to continuous tokenizers. "#Params" is the number of model parameters. "↑" and "↓" indicate whether higher or lower values are better, respectively.

Generalization

We also evaluate reAR on non-standard tokenizers TiTok and AliTok. Unlike RAR, which helps mainly on bidirectional tokenization, reAR consistently improves performance on both bidirectional (TiTok: 4.45 → 4.01) and unidirectional (AliTok: 1.50 → 1.42) tokenizers. Notably, it approaches diffusion-based REPA and outperforms Maskbit while using far fewer parameters (177M vs. 675M/305M).

Table 2: Superior generalization ability. reAR adapts to different tokenizers and achieves state-of-the-art performance with smaller models.

Scaling Effect

We also study if the scaling behavior of the original AR model maintains with reAR. Specifically, we plot the FID under different training epochs for each model size. As Figure 1 shows, the FID consistently decreases as model size and training iteration increase, revealing the potential of reAR on large-scale visual AR models.

Figure 1: Scaling Effect of reAR. As model size increases, the FID at each training step decreases consistently.

Sampling Speed

Like other autoregressive models, reAR benefits from KV-cache to achieve high sampling speed. We measure throughput on a single A800 GPU with batch size 128. With KV-cache, autoregressive models can run much faster than diffusion and MAR. Moreover, reAR-B-AliTok achieves lower FID with faster sampling speed even against parallel-decoding approaches such as Maskbit, TiTok, VAR, and RandAR.

Figure 2: Sampling Speed. Comparison of different methods on FID and throughput (images/sec).

Visual Quality Improvements

reAR demonstrates significant improvements in visual quality. The model generates more coherent and detailed images compared to baseline autoregressive models. The improvements are particularly notable in maintaining consistency across the entire image and reducing artifacts that typically arise from exposure bias.

Citation

@article{he2025rear,
    title={reAR: Rethinking Visual Autoregressive Models via Generator-Tokenizer Consistency Regularization},
    author={Qiyuan He and Yicong Li and Haotian Ye and Jinghao Wang and Xinyao Liao and Pheng-Ann Heng and Stefano Ermon and James Zou and Angela Yao},
    year={2025},
    journal={arXiv preprint},
}