Automating Creativity Assessment in Engineering Design: A Psychometric Validation of AI-Generated Design Problems

Creativity is essential for engineering design, yet its assessment remains challenging due to the resource-intensive nature of traditional evaluation methods. This study investigates the potential of automatic item generation (AIG) using large language models (LLMs) to create psychometrically sound assessment items for measuring creative thinking in engineering. We developed and validated engineering design problems across three domains: ability difference and limitations (e.g., assisting people with learning impairments), transportation and mobility (e.g., reducing traffic congestion in mega cities), and social environments and systems (e.g., improving access to clean water in remote areas). The study comprised three phases with samples matched on race and ethnicity: (1) content validation with a diverse sample of 40 engineers evaluating item clarity and validity, (2) item administration to 462 engineering students, and (3) response evaluation by 65 expert raters assessing originality and effectiveness. Results demonstrated that LLM-generated items achieved comparable or higher content validity rates compared to expert-written items (43% vs. 20% success rate). Bayesian confirmatory factor analysis supported a unidimensional model for fluency, originality, and effectiveness scores, with excellent reliability estimates (ranging from .92 to .95). While fluency showed minimal correlation with originality (r = -.11) and effectiveness (r = -.04), originality and effectiveness demonstrated a strong positive correlation (r = .73). The present research advances our understanding of automated assessment generation in engineering education, provides empirical evidence for the psychometric properties of AI-generated creativity tasks, and offers a scalable approach for measuring creative thinking in engineering classrooms.