Soft-Masked Diffusion Language Models

Michael Hersche, Samuel Moor-Smith, Thomas Hofmann, Abbas Rahimi

Accepted as Conference Paper at the Fourteenth International Conference on Learning Representations 2026 [Paper].

📰 Updates

• [May 2, 2026] Checkpoints for language modeling (OWT) released.
• [Mar 2, 2026] Initial release.

Reproducing Results

Please refer to the Coding and Language folders for experiments with Dream and MDLM, respectively.

Checkpoints Language Modeling

Soft-masking checkpoints for language modeling (OpenWebText) can be found in this Box folder.

Acknowlegement

The coding part was built on top of Dream-7B and the langugae modeling part based on Duo and ReMDM ReMDM.

Citation 📚

If you use the work released here for your research, please consider citing our paper:

@inproceedings{
hersche_softmasking_2026,
title={Soft-Masked Diffusion Language Models},
author={Hersche, Michael and Moor-Smith, Samuel and Hofmann, Thomas and Rahimi, Abbas},
booktitle={The Fourteenth International Conference on Learning Representations (ICLR)},
year={2026},
url={https://openreview.net/forum?id=Gba02UMvrG}
}