MASR: Multi-Agent System with Reflection for the Abstraction and Reasoning Corpus

The Abstraction and Reasoning Corpus (ARC) benchmarksgeneral artificial intelligence, presenting a significant challengeto existing machine learning models and program synthesissolvers due to its focus on broad generalization. In thiswork, we introduce a Multi-Agent System with Reflection(MASR) for ARC. MASR combines Large Language Models(LLMs) and a program synthesis solver based on a DomainSpecific Language (DSL). We analyse the accuracy ofLLMs on ARC and demonstrate unsatisfactory results. Wecreate AugARC, an augmented ARC benchmark, which consistentlyimproves the performance of LLMs compared to thenormal ARC benchmark. Using augmented ARC data, wefine-tune LLMs and observe a significant gain in ARC accuracyafter training. By utilizing reflection, we combine LLMsand a previous DSL solver into our MASR approach for abstractionand reasoning. Our experiments show that MASRoutperforms the previous publicly available ARC systemsthat consist solely of LLMs or DSL solvers, demonstratingthe effectiveness of multi-agent systems on ARC. MASR motivatesresearch to advance previous ARC attempts by combiningthe advantages of LLMs and program synthesis solversinto multi-agent systems.