Abstract

This paper introduces LongBench v2, a benchmark designed to assess theability of LLMs to handle long-context problems requiring deep understandingand reasoning across real-world multitasks. LongBench v2 consists of 503challenging multiple-choice questions, with contexts ranging from 8k to 2Mwords, across six major task categories: single-document QA, multi-document QA,long in-context learning, long-dialogue history understanding, code repositoryunderstanding, and long structured data understanding. To ensure the breadthand the practicality, we collect data from nearly 100 highly educatedindividuals with diverse professional backgrounds. We employ both automated andmanual review processes to maintain high quality and difficulty, resulting inhuman experts achieving only 53.7% accuracy under a 15-minute time constraint.Our evaluation reveals that the best-performing model, when directly answersthe questions, achieves only 50.1% accuracy. In contrast, the o1-preview model,which includes longer reasoning, achieves 57.7%, surpassing the human baselineby 4%. These results highlight the importance of enhanced reasoning ability andscaling inference-time compute to tackle the long-context challenges inLongBench v2. The project is available at https://longbench2.github.io.