Efficient Program Optimization through Knowledge-Enhanced LoRA Fine-Tuning of Large Language Models

Source code optimization enables developers to enhance programs at the human-computer interaction level, thereby improving development efficiency and product quality. With the rise of large language models (LLMs), fine-tuning and prompting have become mainstream solutions for this task. However, both approaches present challenges: fine-tuning is resource-intensive due to the exponential growth in the scale of LLMs, whereas prompting, although resource-efficient, struggles to generate high-quality optimized programs. In this paper, we present CodeOPT, a LoRA-driven approach for fine-tuning LLMs to optimize C/C++ code. Instead of fine-tuning all LLM parameters, CodeOPT leverages LoRA to fine-tune only an optimization adapter, significantly reducing the number of trainable parameters. Additionally, we incorporate prior optimization knowledge during fine-tuning and introduce optimization-based instruction fine-tuning, enabling LLMs to effectively learn from external knowledge sources to improve program optimization. To evaluate the effectiveness of CodeOPT, we benchmarked it against several baselines on challenging programming tasks from different code completion platforms. Experimental results demonstrate that CodeOPT outperforms all baselines, including the state-of-the-art, while keeping modifications to the original program minimal.