Clarifying the Conceptual Landscape in AI Literacy Measurement: A Large Language Model Based Approach

The rapid proliferation of AI literacy instruments has expanded the scope of the field, while introducing diversity in definitions, dimensional structures, and item operationalizations, which complicates cross-instrument comparison and cumulative interpretation. To support a clearer view of this measurement landscape, this study employs a large language model (LLM) based semantic analysis to map the structure of AI literacy content domain across instruments. Drawing on 11 rigorously reviewed AI literacy scales comprising 55 constructs and 272 items, we map construct definitions and item texts into a shared semantic space using transformer-based embeddings. The analysis first examines the coherence and distinctiveness of AI literacy constructs across instruments, and then explores patterns of alignment and divergence between construct labels and their item operationalizations. The results indicate a semantically consolidated core centered on cognitive and technical dimensions of AI literacy, alongside more weakly integrated affective and collaborative components. In addition, patterns of semantic convergence and divergence are observed. Constructs with different labels (e.g., Behavioral Commitment and AI Learning) show similarity in their item operationalizations, whereas other constructs sharing similar labels (e.g., Technical Proficiency and Technical Understanding) diverge in the competencies emphasized by their items. Methodologically, this filed-level mapping provides a foundation for more rigorous instrument development and supports cumulative measurement development. Our results highlight the importance of using clearer and more internally consistent operationalizations to support cumulative progress in measurement development and to enhance the educational applicability of AI literacy frameworks.