Making the wisdom of crowds efficient — with confidence

Efficiently allocating individuals to work on complex decision problems is a key challenge for groups, organizations, and societies. It involves a crucial trade- off: Increasing the number of individuals working on a task typically improves accuracy, but also increases costs. Research in collective intelligence has proposed a plethora of mechanisms to pool the judgments of independent decision makers in order to improve performance. However, these mechanisms are static; because they do not adjust the number of crowd members to the challenge at hand, they incur high, fixed costs for every decision problem. We develop and test three decision rules that make it possible to benefit from the wisdom of the crowd adaptively depending on a case’s difficulty. Our rules rely on decision makers’ confidence judgments to stop crowd growth. Empirical analyses in four real-world domains (cancer diagnoses, false news classification, criminology, and forecasting) using seven datasets show that our adaptive decision rules can result in equal or higher accuracy compared to widely used static crowd aggregators, while relying on fewer individuals. Our findings present easily applicable practical decision guidelines that can substantially boost the efficiency of crowds.