Abstract

We tasked 16 state-of-the-art large language models (LLMs) with estimatingthe likelihood of Artificial General Intelligence (AGI) emerging by 2030. Toassess the quality of these forecasts, we implemented an automated peer reviewprocess (LLM-PR). The LLMs' estimates varied widely, ranging from 3% (Reka-Core) to 47.6% (GPT-4o), with a median of 12.5%. These estimates closely alignwith a recent expert survey that projected a 10% likelihood of AGI by 2027,underscoring the relevance of LLMs in forecasting complex, speculativescenarios. The LLM-PR process demonstrated strong reliability, evidenced by ahigh Intraclass Correlation Coefficient (ICC = 0.79), reflecting notableconsistency in scoring across the models. Among the models, Pplx-70b-onlineemerged as the top performer, while Gemini-1.5-pro-api ranked the lowest. Across-comparison with external benchmarks, such as LMSYS Chatbot Arena,revealed that LLM rankings remained consistent across different evaluationmethods, suggesting that existing benchmarks may not encapsulate some of theskills relevant for AGI prediction. We further explored the use of weightingschemes based on external benchmarks, optimizing the alignment of LLMs'predictions with human expert forecasts. This analysis led to the developmentof a new, 'AGI benchmark' designed to highlight performance differences inAGI-related tasks. Our findings offer insights into LLMs' capabilities inspeculative, interdisciplinary forecasting tasks and emphasize the growing needfor innovative evaluation frameworks for assessing AI performance in complex,uncertain real-world scenarios.