Abstract

Large language models (LLMs), trained on diverse data effectively acquire abreadth of information across various domains. However, their computationalcomplexity, cost, and lack of transparency hinder their direct application forspecialised tasks. In fields such as clinical research, acquiring expertannotations or prior knowledge about predictive models is often costly andtime-consuming. This study proposes the use of LLMs to elicit expert priordistributions for predictive models. This approach also provides an alternativeto in-context learning, where language models are tasked with makingpredictions directly. In this work, we compare LLM-elicited and uninformativepriors, evaluate whether LLMs truthfully generate parameter distributions, andpropose a model selection strategy for in-context learning and priorelicitation. Our findings show that LLM-elicited prior parameter distributionssignificantly reduce predictive error compared to uninformative priors inlow-data settings. Applied to clinical problems, this translates to fewerrequired biological samples, lowering cost and resources. Prior elicitationalso consistently outperforms and proves more reliable than in-context learningat a lower cost, making it a preferred alternative in our setting. Wedemonstrate the utility of this method across various use cases, includingclinical applications. For infection prediction, using LLM-elicited priorsreduced the number of required labels to achieve the same accuracy as anuninformative prior by 55%, 200 days earlier in the study.