A Critical Evaluation of LLMs for Analysis of Perspectives Towards Large-Scale Renewable Energy Projects in the U.S.

Understanding community responses to renewable energy infrastructure siting is essential for accelerating climate action while ensuring a just and equitable energy transition. This study critically examines the capabilities and limitations of large language models (LLMs) for characterizing public sentiment toward renewable energy projects at scale. Drawing on a dataset of 5,095 operational wind and solar projects across the United States, we employed a multi-stage computational methodology to collect and analyze online media coverage, using LLMs to score projects across variables related to opposition types, drivers, and project characteristics. Manually validating a representative sample of the dataset reveals that LLM accuracy varies substantially by variable type: high accuracy (> 95%) for variables with clearly defined, observable indicators but lower accuracy (< 85%) for variables requiring contextual interpretation or narrative synthesis. Analysis across the full dataset indicates that approximately 50% of projects show documented opposition in online media, with threshold effects related to project capacity and differences between wind and solar technologies. However, demographic correlates of opposition documentation appear to reflect digital visibility patterns rather than actual sentiment distributions, raising critical data justice concerns about whose voices become visible in computationally-mediated research. These findings suggest that responsible integration of LLMs into climate and energy social science research requires substantial upfront investment in theoretical grounding, variable operationalization, and validation—meaning effective use of these tools may take considerably longer than anticipated. This performance gradient, where LLMs handle well-defined classification tasks reliably but struggle with tasks requiring contextual judgment and synthesis, has implications beyond research methodology, connecting to broader challenges of scalable oversight in AI systems that will increasingly be asked to assist with the complex, value-laden questions central to climate action. We propose five principles for responsible LLM integration and discuss implications for climate action research methodologies.