Human in the loop chain of code prompting for deterministic tool development with generative AI

This article presents a novel, nested approach to Human-in-the-Loop (HITL) Artificial Intelligence (AI), utilising Chain of Code (CoC) prompting to iteratively develop AI-assisted research tools. Focusing on Generative AI (GenAI) systems such as ChatGPT-4o, this article explores how nested HITL structures—where expert feedback is integrated at each developmental layer—can drive AI outputs to meet domain-specific needs. Through a case study involving a grey literature retrieval tool, this article illustrates how this approach enables researchers to progressively refine GenAI-generated code with modular CoC prompts. Each prompt chain is nested, meaning that outputs from one level serve as inputs to the next, with structured expert feedback guiding refinements at each stage. This tool leverages ChatGPT-4o to generate modular Python scripts for retrieving, filtering, and organising grey literature from targeted Australian government domains.

The nested HITL structure allows GenAI to be continuously aligned with expert-driven goals, resulting in a highly adaptable, transparent, and deterministic research tool. Findings underscore the broader applicability of nested HITL frameworks for complex GenAI-assisted coding tasks, showing how each iterative layer builds upon previous cycles to ensure increasingly precise alignment with researcher requirements. This approach suggests a practical model for GenAI-human collaboration in research, establishing HITL not only as a method for oversight but as a transformative architecture for guiding GenAI outputs through nested, expert-informed feedback loops.