-
Are Discrete Diffusion Models Better Than Auto-regressive Models in Text Generation? Uncovering a Hidden Numerical Issue
With SEDD winning the Best Paper Award at ICML 2024, discrete diffusion models have emerged as a promising contender to auto-regressive models in text generation. In this blog, however, we uncover a hidden yet critical numerical precision issue that negatively impacts generation diversity in discrete diffusion sampling. This flaw highlights the limitations of previous evaluations, which rely solely on the incomplete metric of generative perplexity, resulting in a secretely unfair comparison to auto-regressive models. For complete analyses and proofs, please refer to our paper (http://arxiv.org/pdf/2409.02908).