Abstract

Despite the significant progress in diffusion prior-based image restoration,most existing methods apply uniform processing to the entire image, lacking thecapability to perform region-customized image restoration according to userinstructions. In this work, we propose a new framework, namely InstructRestore,to perform region-adjustable image restoration following human instructions. Toachieve this, we first develop a data generation engine to produce trainingtriplets, each consisting of a high-quality image, the target regiondescription, and the corresponding region mask. With this engine and carefuldata screening, we construct a comprehensive dataset comprising 536,945triplets to support the training and evaluation of this task. We then examinehow to integrate the low-quality image features under the ControlNetarchitecture to adjust the degree of image details enhancement. Consequently,we develop a ControlNet-like model to identify the target region and allocatedifferent integration scales to the target and surrounding regions, enablingregion-customized image restoration that aligns with user instructions.Experimental results demonstrate that our proposed InstructRestore approachenables effective human-instructed image restoration, such as images with bokeheffects and user-instructed local enhancement. Our work advances theinvestigation of interactive image restoration and enhancement techniques.Data, code, and models will be found athttps://github.com/shuaizhengliu/InstructRestore.git.