ECN AI Baseline Index 2024: Stable Accessible AI Performance Across Difficulty Shifts
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
We present the 2024 edition of the ECN AI Baseline Index (EAII), an accessible benchmark evaluating large language models in competitive programming under contest conditions. The 2024 Sapientia–ECN universities division included 11 teams and 16 university-track problems (a 17th, Problem K, was reserved for high-school teams) and introduced a judging platform restricted to C/C++/Pascal. Using a simple C++ prompting protocol with limited feedback rounds, the AI fully solved 13 of 16 problems (81.25%), partially solved two, and failed on one interactive task. Student teams achieved a median of nine and a maximum of twelve solved problems. The resulting 2024 EAII value is 125%. We provide per-problem and team-level analyses, along with figures summarizing score distributions, problem-specific solve rates, and year-over-year comparisons with 2023. Despite a markedly easier contest and a shift in the AI language channel (C++ versus Python), EAII remains stable relative to 2023, supporting the robustness of the normalization procedure. Interpretation is reserved for the Discussion ; this paper focuses on methodology, descriptive analyses, and numerical results.