Abstract

Recent advancements in large vision language models (VLMs) tailored forautonomous driving (AD) have shown strong scene understanding and reasoningcapabilities, making them undeniable candidates for end-to-end driving systems.However, limited work exists on studying the trustworthiness of DriveVLMs -- acritical factor that directly impacts public transportation safety. In thispaper, we introduce AutoTrust, a comprehensive trustworthiness benchmark forlarge vision-language models in autonomous driving (DriveVLMs), consideringdiverse perspectives -- including trustfulness, safety, robustness, privacy,and fairness. We constructed the largest visual question-answering dataset forinvestigating trustworthiness issues in driving scenarios, comprising over 10kunique scenes and 18k queries. We evaluated six publicly available VLMs,spanning from generalist to specialist, from open-source to commercial models.Our exhaustive evaluations have unveiled previously undiscoveredvulnerabilities of DriveVLMs to trustworthiness threats. Specifically, we foundthat the general VLMs like LLaVA-v1.6 and GPT-4o-mini surprisingly outperformspecialized models fine-tuned for driving in terms of overall trustworthiness.DriveVLMs like DriveLM-Agent are particularly vulnerable to disclosingsensitive information. Additionally, both generalist and specialist VLMs remainsusceptible to adversarial attacks and struggle to ensure unbiaseddecision-making across diverse environments and populations. Our findings callfor immediate and decisive action to address the trustworthiness of DriveVLMs-- an issue of critical importance to public safety and the welfare of allcitizens relying on autonomous transportation systems. Our benchmark ispublicly available at \url{https://github.com/taco-group/AutoTrust}, and theleaderboard is released at \url{https://taco-group.github.io/AutoTrust/}.