Abstract

Parameter generation has emerged as a novel paradigm for neural networkdevelopment, offering an alternative to traditional neural network training bysynthesizing high-quality model weights directly. In the context of Low-RankAdaptation (LoRA) for evolving ($\textit{i.e.}$, constantly updated) largelanguage models (LLMs), this approach promises efficient adaptation withoutcostly retraining. However, existing methods face critical limitations insimultaneously achieving scalability and controllability. In this paper, weintroduce $\texttt{ORAL}$, a novel $\textbf{conditional recurrent diffusion}$framework that addresses these challenges. $\texttt{ORAL}$ incorporates a novelconditioning mechanism that integrates model architecture and textual taskspecifications, enabling the generation of task-specific LoRA parameters thatcan seamlessly transfer across evolving foundation models. Our approachsuccessfully scales to billions-of-parameter LLMs and maintainscontrollability. Through extensive experiments across seven language tasks,four vision tasks, and three multimodal tasks using five pre-trained LLMs, wedemonstrate that $\texttt{ORAL}$ generates high-quality LoRA parameters thatachieve comparable or superior performance to vanilla trained counterparts.