Abstract

Large language models (LLMs) have shown significant multilingualcapabilities. However, the mechanisms underlying the development of thesecapabilities during pre-training are not well understood. In this paper, we usecode LLMs as an experimental platform to explore the evolution of multilingualcapabilities in LLMs during the pre-training process. Based on ourobservations, we propose the Babel Tower Hypothesis, which describes the entireprocess of LLMs acquiring new language capabilities. During the learningprocess, multiple languages initially share a single knowledge system dominatedby the primary language and gradually develop language-specific knowledgesystems. We then validate the above hypothesis by tracking the internal statesof the LLMs through identifying working languages and language transferringneurons. Experimental results show that the internal state changes of the LLMare consistent with our Babel Tower Hypothesis. Building on these insights, wepropose a novel method to construct an optimized pre-training corpus formultilingual code LLMs, which significantly outperforms LLMs trained on theoriginal corpus. The proposed Babel Tower Hypothesis provides new insights intodesigning pre-training data distributions to achieve optimal multilingualcapabilities in LLMs.