Abstract

Large language models (LLMs) perform well in medical QA, but theireffectiveness in Japanese contexts is limited due to privacy constraints thatprevent the use of commercial models like GPT-4 in clinical settings. As aresult, recent efforts focus on instruction-tuning open-source LLMs, though thepotential of combining them with retrieval-augmented generation (RAG) remainsunderexplored. To bridge this gap, we are the first to explore a knowledgegraph-based (KG) RAG framework for Japanese medical QA small-scale open-sourceLLMs. Experimental results show that KG-based RAG has only a limited impact onJapanese medical QA using small-scale open-source LLMs. Further case studiesreveal that the effectiveness of the RAG is sensitive to the quality andrelevance of the external retrieved content. These findings offer valuableinsights into the challenges and potential of applying RAG in Japanese medicalQA, while also serving as a reference for other low-resource languages.