Do humans learn like transformer networks?

Do humans learn like transformers? We trained both humans and transformer networks on a rule learning task where they had to respond to a query at the end of a sequence of symbols (the context). At test, we measured both “in-context” learning (the ability to generalise the rule to novel queries) and “in-weights” learning (the recall of past experiences from memory). Manipulating the diversity and redundancy of examples in the training distribution, we found that humans and transformers respond in very similar ways. In both types of learner, redundancy and diversity trade off in driving in-weights and in-context learning respectively, whereas a composite distribution that includes a balanced mix of redundancy and diversity allows the two strategies to be used in tandem. However, we also found that while humans benefit from dynamic training schedules that emphasise diverse examples early on, transformers do not. So, whilst the same data distributional properties promote learning in humans and transformer networks, only people benefit from curricula.