Chatbot-guided Search delivers Low-Relevance News and can exacerbate Gender Gaps in Political Knowledge

Policymakers and researchers are increasingly concerned with how algorithmic systems shape public access to political information. AI-powered chatbots, such as OpenAI’s GPT-4o, are being adopted as search tools, yet their impacts on information quality and user reasoning remain unclear. In this study, we present a novel, controlled framework for evaluating LLM-mediated political informa- tion search using a custom-built chatbot. Unlike prior work, our design permits direct comparison across LLMs and traditional sources (e.g., Google News), enabling systematic and repeatable evaluations of response quality, bias, and knowledge gain. Using this framework, we supplemented an audit of 592,008 arti- cles retrieved in response to 80,640 political queries across 11 countries, with a conducted a bilingual survey experiment (N = 922) in the U.S. and India. Partic- ipants were randomly assigned to four retrieval formats, yielding 14,564 messages across 1,522 chatbot conversations. We find that GPT-4o retrieves less relevant and more controversial content than Google News. Despite this, GPT-4o users reported higher knowledge gains but demonstrated lower misinformation discern- ment—particularly in chatbot formats. These effects were moderated by gender, with non-male participants gaining more knowledge but showing greater declines in accuracy. Our findings highlight trade-offs between perceived learning and critical evaluation in AI-curated political information, underscoring the need for transparent, trustworthy design in emerging conversational systems.