A Markov Chain Monte Carlo Procedure with Simulated Annealing for Shortening Assessments

Symptom screeners, patient assessments, and cognitive tests are used to measure constructs such as health-related quality of life, levels of mental health symptoms, and ability. The items are often summarized, through a simple unweighted sum score or through more advanced methods, including factor analysis, Rasch modeling, and item response theory. However, long instruments increase respondent burden and it may be desirable to shorten them. One consideration that is often overlooked is that in many contexts, these instruments are used less for precise measurement of a latent construct but instead for accurate prediction of a diagnosis, expert judgment, or classification. As such, we present a method to shorten instruments that prioritizes reproduction of expert judgment based on an unweighted sum score through the use of a Markov Chain Monte Carlo algorithm with simulated annealing to explore the search space of potential short forms. This method is orders of magnitude more efficient than brute force search and has the advantage of optimizing for prediction of the desired outcome directly, instead of optimizing on the measurement of an intermediate outcome that may not be perfectly aligned with the decision criteria of interest. We prove the correctness of the algorithm, demonstrate its effectiveness and efficiency through simulation studies, and apply it to a screener for alcohol use disorder, demonstrating that under the unweighted sum score scoring constraint, shortened forms can more accurately predict expert judgments than full forms.