Abstract

Tool-calling has changed Large Language Model (LLM) applications byintegrating external tools, significantly enhancing their functionality acrossdiverse tasks. However, this integration also introduces new securityvulnerabilities, particularly in the tool scheduling mechanisms of LLM, whichhave not been extensively studied. To fill this gap, we present ToolCommander,a novel framework designed to exploit vulnerabilities in LLM tool-callingsystems through adversarial tool injection. Our framework employs awell-designed two-stage attack strategy. Firstly, it injects malicious tools tocollect user queries, then dynamically updates the injected tools based on thestolen information to enhance subsequent attacks. These stages enableToolCommander to execute privacy theft, launch denial-of-service attacks, andeven manipulate business competition by triggering unscheduled tool-calling.Notably, the ASR reaches 91.67% for privacy theft and hits 100% fordenial-of-service and unscheduled tool calling in certain cases. Our workdemonstrates that these vulnerabilities can lead to severe consequences beyondsimple misuse of tool-calling systems, underscoring the urgent need for robustdefensive strategies to secure LLM Tool-calling systems.