System Identification with Large Language Models as Evolutionary Algorithms

In this article, an emergent and promising algorithm, Large Language Models as Evolutionary Algorithms (LMEA), is investigated for applicability to System Identification (SI). The idea of using Large Language Models (LLMs) for optimisation is extended to simulated training environments, ready for engineering-related problems. A variation of LMEA, Real Value Optimised Large Language Models as Evolutionary Algorithms (RVOLMEA) is proposed to address the limitations in computer program parameterisation within LMEA. Leveraging Real-valued Optimisation (RVO) within the LMEA framework, RVOLMEA realises the value of variables used within evolution before testing. Parameterisation performed as such is shown to reduce the amount of queries required to achieve convergence, increasing the quality of produced programs. Results are analysed through the Mean Squared Error (MSE) and Wasserstein Distance of derived functions extracted from the simulator. Included are investigations against both simple and complex tasks for SI. Comparative analysis is performed against both the original LMEA technique and neural networks. It is shown that for the task at hand, RVOLMEA achieves neural network-level performance, effectively discovering the actual system dynamics. The testing environments for the discovery of the system dynamics are a UAV simulator and a Gymnasium environment. The code for implementing RVOLMEA on the complex problem is available at the following GitHub repository: here.