Can Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) Generate New Knowledge for Urban Studies?

Retrieval-Augmented Generation (RAG) has demonstrated promising potential in addressing domain-specific challenges by integrating up-to-date, expert-level knowledge into the outputs of Large Language Models (LLM). However, optimizing these models for urban studies and other specialized domains remains a key challenge. This paper investigates the application of RAG-enhanced LLMs in the context of Urban Vacant Land (UVL), a pressing global issue for sustainable urban development. Drawing on a Systematic Literature Review (SLR) of UVL-related research retrieved from the Web of Science (WOS) database, we constructed a research knowledge base to support RAG-based enhancements. We then designed five challenging prompts focused on UVL, each targeting areas where current academic literature remains limited, aiming to assess the models' ability to generate new knowledge beyond existing findings. Through a combination of automated and human evaluations, we found that RAG significantly improved LLM performance in addressing specialized topics. However, limitations persist in the models’ capacity for knowledge generation. Most outputs were limited to proposing related research questions and preliminary experimental designs. These findings underscore both the potential and the current boundaries of RAG-enhanced LLMs in supporting theoretical innovation and policy-oriented research within complex urban domains.