Conversational Interest between Teachers and Second Language Learners Evaluated by Humans and LLMs

Stimulating language learners’ interest and engagement is essential to successful second languageacquisition, but it can be hard to translate this intuition into effective learning resources. Wecarried out the first quantitative investigation into the linguistic and pragmatic features that makean educational conversation interesting. To do so, we collected a new corpus of human interestratings for conversations between teachers and second language learners of English. Using thiscorpus, we showed that concreteness, comprehensibility (assessed via a new computationalmetric, the GIS score) and uptake (i.e., the degree to which a teacher and a student’s turn build onone another) all have unique relations to interest. This provides proof of concept that - despite thehigh degree of subjectivity involved in perceptions of interest - it is possible to extract featuresthat make a conversation interesting for the average learner. We also showed that - when a newtopic is introduced in a conversation - raters judge the corresponding turn as being moreinteresting than they expected, suggesting a role for expectations in determining interest. Finally,we evaluated the capability of Large Language Models (LLMs) to align with human ratings ofconversational interest, and showed that - while the task is difficult for all models - those modelsthat are fine-tuned on human preferences do better than those that are not. These findings provideinsights into what makes for interesting conversations between teachers and second-languagelearners of English, and lay the foundations for future work on the optimization of LLMs formore engaging language learning interactions.