Assessing Vocational Interests through Chat: Development and Validation of the Career Guidance Chatbot (CGC-bot)

The current research developed the Career Guidance Chatbot (CGC-bot) that collects high-quality, diverse work preference information from users, and the subsequent twenty Natural Language Processing (NLP) + Machine Learning (ML) algorithm-powered machines that predict basic interest scores from user’s chat with the CGC-bot, and searches for top O*NET occupations based on user’s basic interest profile prediction. We evaluate the consistency and performance of text embeddings generated from various NLP models as predictors for basic interest scores, and we evaluate the psychometric properties of machine-predicted interest scores. 1250 U.S. adults and 328 college students were recruited from Prolific and the course credit participant pool (SONA) at a large midwestern university to complete a two-part study that is spaced one week apart, where one part is chatting with the CGC-bot and the other is to complete a basic interest survey. A total of 1176 participants finished both parts of the study and were included in our analyses. Results suggested that there is good consistency across interest score prediction patterns from machines built on various groups of text embeddings, and machine-predicted scores shared comparable psychometric properties as self-reported scores. On the other hand, machine-predicted and self-report interest scores showed lower monotrait-heteromethod correlations (r = .20 - .54) than those found between interest inventories. It is also notable that participants enjoyed the chatbot less due to its higher mental demand and longer interaction time.Together, these tools integrate recent advancements in technology with vocational interest research for improving real-world applications of interest measurement and the study sets a benchmark for future studies that aim to measure vocational interests with chat-based text data.