Valid and reliable? Basic criteria and modern methods for assessing psychometric quality

Improving the validity, reliability, and replicability of research in the social and health sciences depends on the quality of measurement. We argue that judging whether the sum score of a multi-item measure adequately represents the intended latent variable rests on four basic criteria — unidimensionality, local independence of items, ordered response categories, and invariance — which can serve as a minimal checklist for end users assessing the quality of a psychometric study. A central recommendation is to stop interpreting model and item fit against rule-of-thumb critical values and instead use simulation-based cutoffs tailored to the data at hand. Because most published psychometric work relies on rule-of-thumb cutoffs, established scales are in urgent need of re-analysis with appropriate methods. We provide an up-to-date overview of methods for evaluating each criterion within Rasch Measurement Theory, point to freely available software that makes these methods accessible, and note outdated approaches to avoid. A draft reporting guideline is provided, as we believe that a consensus-based reporting standard may facilitate more transparent evaluations of scales. Finally, we argue for a higher bar for calling a measure "valid and reliable": the existence of an original validation paper, often old and based on outdated methods, is not in itself sufficient evidence of validity.