Text-to-Music Generation via Structured Captions
and Interpretable Evaluation

Abstract

Text descriptions of music are central to the performance of text-to-music generation (TTM) methods, but creation of paired text information for music generation is relatively underexplored. Here, we propose training latent diffusion/flow-based music generation models conditioned on structured captions (SCAPs) and compare with typical free-form captions via a set of controlled ablations. We compare free-form vs. structured captions, partial vs. full information coverage, and global vs. temporal text information during training, by fine-tuning different configurations of a pretrained flow-based diffusion transformer (DiT) and using a multi-stage inference-time prompt re-writer. Further, we introduce a new text-alignment evaluation protocol inspired by Davidsonian Scene Graphs (DSGs) for a fine-grained, interpretable measure of faithfulness of TTM methods. With multiple metrics, we show 1) that fine-tuning our DiT on SCAPs significantly improves text-alignment on music generation outputs while also improving quality, and 2) our proposed Music DSG metric better correlates with human preference than CLAP.

BibTeX

@article{wu_scap,
  title   = {Text-to-Music Generation via Structured Captions and Interpretable Evaluation},
  author  = {Wu, Junkai and Wang, Zhepei and Zhu, Ge and Lee, Sarah Hye-yeon and
             Ostendorf, Mari and Bryan, Nicholas J.},
  journal = {arXiv preprint arXiv:2611.00685},
  year    = {2026},
  url     = {https://arxiv.org/abs/2611.00685}
}