Abstract

Retrieval-Augmented Generation (RAG) mitigates hallucinations in LargeLanguage Models (LLMs) by integrating external knowledge. However, conflictsbetween parametric knowledge and retrieved context pose challenges,particularly when retrieved information is unreliable or the model's internalknowledge is outdated. In such cases, LLMs struggle to determine whether torely more on their own parameters or the conflicted context. To address this,we propose **CK-PLUG**, a plug-and-play method for controlling LLMs' relianceon parametric and contextual knowledge. We introduce a novel knowledgeconsistency metric, Confidence Gain, which detects knowledge conflicts bymeasuring entropy shifts in token probability distributions after contextinsertion. CK-PLUG then enables fine-grained control over knowledge preferenceby adjusting the probability distribution of tokens with negative confidencegain through a single tuning parameter. Experiments demonstrate CK-PLUG'sability to significantly regulate knowledge reliance in counterfactual RAGscenarios while maintaining generation fluency and knowledge accuracy. Forinstance, on Llama3-8B, memory recall (MR) of RAG response can be adjustedwithin a broad range (9.9%-71.9%), compared to the baseline of 42.1%. Moreover,CK-PLUG supports adaptive control based on the model's confidence in bothinternal and external knowledge, achieving consistent performance improvementsacross various general RAG tasks. Our code is available at:$\href{https://github.com/byronBBL/CK-PLUG}{\text{this https URL}}$.