Abstract

To address the bottleneck of accurate user intent interpretation within thecurrent video generation community, we present Any2Caption, a novel frameworkfor controllable video generation under any condition. The key idea is todecouple various condition interpretation steps from the video synthesis step.By leveraging modern multimodal large language models (MLLMs), Any2Captioninterprets diverse inputs--text, images, videos, and specialized cues such asregion, motion, and camera poses--into dense, structured captions that offerbackbone video generators with better guidance. We also introduce Any2CapIns, alarge-scale dataset with 337K instances and 407K conditions forany-condition-to-caption instruction tuning. Comprehensive evaluationsdemonstrate significant improvements of our system in controllability and videoquality across various aspects of existing video generation models. ProjectPage: https://sqwu.top/Any2Cap/