Reinforcement Learning versus Natural Language Programs: Where is Flexible Planning and Problem Solving in Natural Intelligence Coming From?

Flexible planning and problem solving are hallmarks of intelligence. Cognitive computational neuroscientists propose that reinforcement learning (RL) is the most-likely candidate as the computational substrate of flexible behaviors. Considerable experimental effort has been dedicated in identifying the biophysical correlate of RL algorithms in animals and humans. An important assumption in these endeavors is that animals and humans solve complex tasks in the shared computational way. We argue that model-based reinforcement learning is not a very good framework for intelligent behaviors in both animals and humans. Our argument is that while animals might solve complex problems in non-linguistic algorithmized form, such as model-free RL, humans utilize language as expressible heuristics to reason and solve problems. Recently, roboticists show that using natural language programs as the computational substrate is the secret weapon for solving difficult long-horizon planning problems in robots, such as making a cup of tea. In this paper, we argue 1. natural language program is the computational substrate for humans and artificial agents, such as Large Language Models, solving complex problems and tasks; 2. model-free reinforcement learning (RL) and its variants is the most probable computational substrate for complex flexible behaviors in animals.