The Human Oversight Fallacy–and What We Can Do About It

Human oversight, as one of several ethical principles required for the safe use of AI systems, plays a prominent role in the European Union's Artificial Intelligence Act (AIA). This position paper argues that, while human oversight remains a necessary safeguard to mitigate the risks posed by AI systems aimed at directly interacting with humans across diverse domains, its current conceptualization rests on problematic assumptions about human judgment, epistemic access, and institutional responsibilities. Drawing on empirical findings from psychology and human–AI interaction research, we show that human oversight, as presently envisaged, runs risk to remaining an illusion of control if the conditions for meaningful oversight are not made explicit. To address these limitations, we propose a conceptual reorientation of human oversight away from reactive approval and toward structurally embedded control, by suggesting incentive-based ethical stress testing, the establishment of normative clarity, and the systematic accounting for the limits of human oversight. We conclude that only by grounding oversight in realistic assumptions about human cognition and institutional incentives can it fulfill its intended role in ensuring trustworthy AI.