Abstract

Large Language Models (LLMs) have shown tremendous potential as agents,excelling at tasks that require multiple rounds of reasoning and interactions.Rejection Sampling Fine-Tuning (RFT) has emerged as an effective method forfinetuning LLMs as agents: it first imitates expert-generated successfultrajectories and further improves agentic skills through iterative fine-tuningon successful, self-generated trajectories. However, since the expert (e.g.,GPT-4) succeeds primarily on simpler subtasks and RFT inherently favors simplerscenarios, many complex subtasks remain unsolved and persistentlyout-of-distribution (OOD). Upon investigating these challenging subtasks, wediscovered that previously failed expert trajectories can often providevaluable guidance, e.g., plans and key actions, that can significantly improveagent exploration efficiency and acquisition of critical skills. Motivated bythese observations, we propose Exploring Expert Failures (EEF), whichidentifies beneficial actions from failed expert trajectories and integratesthem into the training dataset. Potentially harmful actions are meticulouslyexcluded to prevent contamination of the model learning process. By leveragingthe beneficial actions in expert failures, EEF successfully solves somepreviously unsolvable subtasks and improves agent tuning performance.Remarkably, our approach achieved a 62\% win rate in WebShop, outperforming RFT(53. 6\%) and GPT-4 (35. 6\%), and to the best of our knowledge, setting a newstate-of-the-art as the first method to surpass a score of 0.81 in WebShop andexceed 81 in SciWorld.