Design and Evaluation of a Research-Guided Chatbot for Novice Researchers

(1) Background: Evaluating popular educational chatbots (EC) shows that existing EC impacts users' ability to synthesize and take ownership of their work, potentially leading to misinformation. This study aims to design and evaluate a research-guided chatbot that improves research synthesis. (2) Method: One hundred sixty-five graduate students evaluated the chatbot. The qualitative data were analysed using natural language processing for sentiment analysis and emotion detection. (3) Results: The findings show that "interesting" and "enjoyable" were frequent words used in user feedback. Sentiment analysis findings show that 47% suggested users are satisfied with the chatbot's recommendation potential; 46.3% found the guided chatbot to be the most effective in simplifying research methodology concepts; 48.4% identified the chatbot as the most effective tool for conceptualizing research questions and hypotheses and excellent in fostering ownership of their ideas; 85.3% rated the chatbot as fostering creativity; and 81.0% rated the chatbot positively for its ability to minimize misinformation. Two themes were identified in the thematic analysis: attitudes and improvements to future research-guided chatbots, emphasizing simplicity. (4) Conclusions: The strong preference for the guided chatbot underlines the potential for future improvements in EC with the flexibility to gain ownership of work with a guided approach.