Language Model-Based Analysis of Teaching: Potential and Limitations in Evaluating High-Level Instructional Skills

The automated evaluation of teaching effectivenessusing Natural Language Processing (NLP) is explored, with aspecific focus on the capacity of pre-trained language models toassess high-inference instructional practices. Datasets comprisingK-12 classroom transcripts and simulated teaching scenariosare utilised to evaluate model performance. Methodologicalinnovations include a two-stage approach to handle lengthy andnoisy textual data, as well as techniques to mitigate the impact ofskewed rating distributions. Results reveal that language modelsdemonstrate varying degrees of success in capturing differentfacets of instructional quality, with performance influenced bythe level of inference required. The study contributes to understanding both the promise and the current limitations of applyinglanguage models to the complex task of evaluating teaching.