A Replication & Reproduction of "Time-dependent Evaluation of Recommender Systems"

Standard evaluation of recommender systems typically uses static, single-number metrics, which fail to capture how algorithm performance evolves over time. This paper addresses this research problem by replicating and extending the work of Scheidt and Beel, who proposed a time-dependent evaluation approach. Following their methodology, we first replicated their experiments on three datasets and then applied the time-dependent analysis to five additional datasets and three new algorithms, focusing on the nDCG metric. Our results confirm the original findings: algorithm performance and rankings are highly unstable and change significantly over time, particularly in a dataset’s early stages, with no single algorithm proving universally superior. Although the study was limited by computational costs, our findings strongly support the conclusion that a time-aware evaluation is crucial for a more realistic and exact assessment of recommender systems, moving beyond static benchmarks.