Abstract

Large Language Models (LLMs) have recently shown remarkable ability toprocess not only text but also multimodal inputs such as speech and audio.However, most existing models primarily focus on analyzing input signals usingtext instructions, overlooking scenarios in which speech instructions and audioare mixed and serve as inputs to the model. To address these challenges, weintroduce Solla, a novel framework designed to understand speech-basedquestions and hear the acoustic context concurrently. Solla incorporates anaudio tagging module to effectively identify and represent audio events, aswell as an ASR-assisted prediction method to improve comprehension of spokencontent. To rigorously evaluate Solla and other publicly available models, wepropose a new benchmark dataset called SA-Eval, which includes three tasks:audio event classification, audio captioning, and audio question answering.SA-Eval has diverse speech instruction with various speaking styles,encompassing two difficulty levels, easy and hard, to capture the range ofreal-world acoustic conditions. Experimental results show that Solla performson par with or outperforms baseline models on both the easy and hard test sets,underscoring its effectiveness in jointly understanding speech and audio.