Abstract

Large language models (LLMs) have recently transformed from text-basedassistants to autonomous agents capable of planning, reasoning, and iterativelyimproving their actions. While numerical reward signals and verifiers caneffectively rank candidate actions, they often provide limited contextualguidance. In contrast, natural language feedback better aligns with thegenerative capabilities of LLMs, providing richer and more actionablesuggestions. However, parsing and implementing this feedback effectively can bechallenging for LLM-based agents. In this work, we introduce Critique-GuidedImprovement (CGI), a novel two-player framework, comprising an actor model thatexplores an environment and a critic model that generates detailed naturelanguage feedback. By training the critic to produce fine-grained assessmentsand actionable revisions, and the actor to utilize these critiques, ourapproach promotes more robust exploration of alternative strategies whileavoiding local optima. Experiments in three interactive environments show thatCGI outperforms existing baselines by a substantial margin. Notably, even asmall critic model surpasses GPT-4 in feedback quality. The resulting actorachieves state-of-the-art performance, demonstrating the power of explicititerative guidance to enhance decision-making in LLM-based agents.