Abstract

Large Language Model (LLM) based coding tools have been tremendouslysuccessful as software development assistants, yet they are often designed forgeneral purpose programming tasks and perform poorly for more specializeddomains such as high performance computing. Creating specialized models andtools for these domains is crucial towards gaining the benefits of LLMs inareas such as HPC. While previous work has explored HPC-specific models, LLMsstill struggle to generate parallel code and it is not at all clear whathurdles are still holding back these LLMs and what must be done to overcomethem. In this work, we conduct an in-depth study along the many axes offine-tuning a specialized HPC LLM in order to better understand the challenges.Based on our findings we fine-tune and evaluate a specialized HPC LLM that isshown to be the best performing open-source code LLM for parallel codegeneration to date.