Realistic Simulation of Item Difficulties

Simulation studies are commonly used to improve understanding of psychometric models. For many common models, an essential feature of the simulation is the relative (to variation in person parameters) variation in item difficulties. A common practice has been to generate both item difficulty parameters and person parameters directly from a standard normal distribution or to otherwise stipulate distributions for both parameters at the outset of the simulation. This approach implicitly assumes that these distributions can adequately represent empirical data—an assumption that warrants careful examination. In this paper, leveraging 73 datasets from the Item Response Warehouse (Domingue et al., 2023), we examine the variability of item difficulty distributions in real-world datasets and investigate how this variation influences estimation and simulation. We identify key distributional characteristics (e.g., variance and skewness) and propose a new method for simulating realistic item difficulties based on empirical data. This method enhances the realism and applicability of simulation results, making them more reflective of real-world measurement conditions and improving the robustness of psychometric model evaluation