Humans actively sample evidence to support prior beliefs
Curation statements for this article:-
Curated by eLife
Evaluation Summary:
Kaanders et al. investigate how the sampling of visual information by human subjects is biased toward their previous choice. The novel experiments and rigorous analyses largely support the presence of a 'confirmation bias' when information sampling is under the subjects' control. After ruling out some remaining alternative explanations of the observed behavior the paper will be of broad interest in cognitive neuroscience.
(This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #2 agreed to share their name with the authors.)
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (eLife)
- Neuroscience (eLife)
Abstract
No one likes to be wrong. Previous research has shown that participants may underweight information incompatible with previous choices, a phenomenon called confirmation bias. In this paper, we argue that a similar bias exists in the way information is actively sought. We investigate how choice influences information gathering using a perceptual choice task and find that participants sample more information from a previously chosen alternative. Furthermore, the higher the confidence in the initial choice, the more biased information sampling becomes. As a consequence, when faced with the possibility of revising an earlier decision, participants are more likely to stick with their original choice, even when incorrect. Critically, we show that agency controls this phenomenon. The effect disappears in a fixed sampling condition where presentation of evidence is controlled by the experimenter, suggesting that the way in which confirmatory evidence is acquired critically impacts the decision process. These results suggest active information acquisition plays a critical role in the propagation of strongly held beliefs over time.
Article activity feed
-
-
Evaluation Summary:
Kaanders et al. investigate how the sampling of visual information by human subjects is biased toward their previous choice. The novel experiments and rigorous analyses largely support the presence of a 'confirmation bias' when information sampling is under the subjects' control. After ruling out some remaining alternative explanations of the observed behavior the paper will be of broad interest in cognitive neuroscience.
(This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #2 agreed to share their name with the authors.)
-
Reviewer #1 (Public Review):
This study aims at understanding whether and how the sequential sampling of information during decision-making is influenced by previous choices. For this purpose, the authors have designed a novel two-step numerosity discrimination paradigm for which, in every trial, human subjects compare twice the number of dots contained in two areas: once during a two-interval forced choice, and a second time after a free sampling period during subjects can choose which area to observe (only one area being visible at a time). Across two experiments, the authors describe a 'confirmation bias' in information sampling toward the initial choice, and show that this sampling bias is only present when participants are in control of information sampling. This is an intriguing and novel finding that the authors describe in terms …
Reviewer #1 (Public Review):
This study aims at understanding whether and how the sequential sampling of information during decision-making is influenced by previous choices. For this purpose, the authors have designed a novel two-step numerosity discrimination paradigm for which, in every trial, human subjects compare twice the number of dots contained in two areas: once during a two-interval forced choice, and a second time after a free sampling period during subjects can choose which area to observe (only one area being visible at a time). Across two experiments, the authors describe a 'confirmation bias' in information sampling toward the initial choice, and show that this sampling bias is only present when participants are in control of information sampling. This is an intriguing and novel finding that the authors describe in terms of an 'economic model' of information sampling, which nevertheless requires additional analyses to rule out alternative accounts of the observed behavior.
The authors characterize the sampling bias toward the initial choice as a form of confirmation bias - i.e., the tendency to underweight information incompatible with previous choices (which is how this type of bias is identified in earlier work). However, this interpretation of the observed sampling bias is only valid if we assume that subjects aim at sampling the area which contains the largest perceived number of dots. This is what is proposed by the 'economic model' of sampling described by the authors in the Supplemental Material, but this sampling strategy is suboptimal for the task carried out by the subjects, and is not compared against alternative sampling strategies proposed in the existing literature. In the (free) sampling period, rather than sampling from the area which contains the largest number of dots, an optimal agent should aim at reducing the uncertainty regarding which area contains the largest number of dots. Given that the task is difficult (based on the psychometric curves shown in Figure 1B and 3B), subjects are uncertain about the exact number of dots present in both areas, and they should thus theoretically aim at reducing the uncertainty about the difference in number of dots between the two areas, not only the uncertainty about the area with the largest perceived number of dots at the time of the initial choice. Existing literature has shown that human subjects engage in this kind of 'directed exploration', but this sampling strategy (which is the optimal strategy in the task) is currently not considered in the main text, and not compared to the suboptimal strategy assumed by the 'economic model' described in the Supplemental Material. It should be possible to compare the two sampling strategies through model comparison, based on existing models of numerosity perception/discrimination. Even if the two sampling strategies cannot be formally compared, the authors should explicitly consider the uncertainty-minimizing strategy as an alternative to their current model, and discuss this possibility in the discussion. While the idea that the observed sampling bias reflects a confirmation bias is appealing, it is important that the authors state explicitly how the sampling strategy assumed by their model (i.e., sampling from the area with the largest perceived number of dots) is critical to link sampling bias to confirmation bias.
Related to point 1, the manuscript lacks details regarding how behavior is rewarded in the task. If only the second (and first) choice(s) are used to determine task performance (e.g., monetary reward), then the optimal strategy during the sampling phase is really to reduce uncertainty about the difference in number of dots between the two areas. By contrast, if the amount of time sampling the area with the largest number of dots influences task performance, then the optimal strategy may resemble the one assumed by the model described by the authors in the Supplemental Material. In relation to the proposed model and its sampling strategy, it would be very useful to provide further information about the incentivization strategy used to motivate subjects in the task. The authors report in the Methods (L563) that "participants received no feedback about the correctness of their choice", but it is unclear why they chose not to provide trial-to-trial feedback. Another aspect of the task that lacks details in the manuscript is the distribution of the number of dots in the two areas across trials. I assume that the reference point (the mean number of dots across the two areas) is not fixed across trials, so that subjects have to sample both areas to determine which one contains more dots. But if it is indeed the case, then the sampling bias reported by the authors does not necessarily indicate a confirmation bias, but rather which area is associated with the largest perceived uncertainty regarding its number of dots. Details about distributions of the number of dots in the two areas (the one containing more dots and the one containing less dots) should be provided in the revised manuscript.
Regarding the contrast between active and passive information sampling, the absence of sampling bias in the passive (fixed) sampling condition - which the authors interpret as an absence of confirmation bias - deserves further control analyses. First, it would be very useful to know whether the overall fraction of changes-of-mind is significantly lower in the active (free) sampling condition relative to the passive (fixed) sampling condition. Indeed, as described by the authors, the confirmation bias indicates a "tendency to underweight information incompatible with previous choices", and it is therefore expected that subjects should overall change their minds during the second choice less often when this bias is expressed than when it is not. Figure 4D does not provide this information, and it would be very useful to offer in Figure Supplement 4 an alternative version of Figure 4D where the x-axis corresponds to the proportion of time where the chosen area is sampled, normalized not by the total time but by the time any area is sampled (excluding the time spent away from the two options). Finally, if the effect of sampling bias on changes-of-mind is shown to disappear in the fixed condition, another control effect - not currently tested as far as I can tell - should be present in both conditions. Given that the task is difficult (based on the psychometric curves shown in Figure 1B and 3B), the objective accuracy of the second choice should be maximal when the two areas are sampled equally during the sampling phase, and decrease monotonically as a function of the sampling bias (in either direction). This should be true irrespective of the sampling condition (free or fixed), and would confirm that the sampling bias (interpreted by the authors as a confirmation bias) indeed has a cost on accuracy (due to the underweighting of information incompatible with previous choices).
-
Reviewer #2 (Public Review):
In this manuscript the authors ask whether active information sampling is biased towards a previously chosen alternative. In two perceptual choice experiments, the authors show that, following an initial choice, human participants tend to sample more information from the chosen alternative, which in turn biases a second (final) choice. This sampling bias is magnified when the initial decision is made with higher confidence; and dissipates when, post-choice, extra information is passively perceived than actively sampled. Taken together these findings speak in favour of a "confirmation bias" governing active information sampling. This finding extends previous studies that have identified forms of confirmation biased in perceptual decisions. The experimental protocol and analyses are appropriate and rigorous.
Th…
Reviewer #2 (Public Review):
In this manuscript the authors ask whether active information sampling is biased towards a previously chosen alternative. In two perceptual choice experiments, the authors show that, following an initial choice, human participants tend to sample more information from the chosen alternative, which in turn biases a second (final) choice. This sampling bias is magnified when the initial decision is made with higher confidence; and dissipates when, post-choice, extra information is passively perceived than actively sampled. Taken together these findings speak in favour of a "confirmation bias" governing active information sampling. This finding extends previous studies that have identified forms of confirmation biased in perceptual decisions. The experimental protocol and analyses are appropriate and rigorous.
The overall conclusion is supported by the key analyses performed. To further strengthen the paper and corroborate the conclusion more methodological details and additional analysis need to be performed.
Some details concerning the perceptual task are missing. It is not clear how the dot patches were delivered to the participants. Fig. 1A shows two stimulus presentation phases but the details on the duration of each phase, or whether there was a gap between phases, are missing.
The seemingly sequential presentation of the two choice-stimuli invites questions about temporal order biases, both in terms of the initial choice and in terms of the subsequent sampling (how do decision weights change across time). Do people show a choice and a sampling bias towards the first/ last presented alternative? The confirmation bias effect could occur spuriously if, say, people tend to choose the early alternative and tend to also sample from the early presented alternative due to having forgotten that information.
Influential theories of information sampling posit that sampling aims at reducing uncertainty. Theories in numerical cognition also hold that representational noise scales positively with numerosity. Linking the two, could it be the case that participants happen to choose more often (following the task instructions) & sample more from the alternative with the largest number dots (just because it is associated with higher uncertainty)? A task in which the subjects were instructed to select the alternative with the fewer dots could help, although the authors could rule out this alternative hypothesis by relying on the confidence data.
Experiment 2 draws a distinction between passive and active post-choice sampling. In the "passive" session the first choice seems to be less meaningful given that the subjects know that they will subsequently observe more evidence. It would be important to show the psychometric data in Figure 2B separately for active/ passive sampling and examine if there are any differences between the two sessions (e.g. Choice 1 in fixed sampling might be characterised by lower accuracy).
Highlighting the chosen alternative with a green arrow could bias the subsequent post-choice sampling towards the chosen alternative.
It is not clear why feedback was not provided on a trial-by-trial basis. More generally, how were participants incentivised? Did they have reasons to assign equal importance to the first and second choices?
-
Reviewer #3 (Public Review):
Kaanders et al investigated how information sampling is biased by human agents to support their previous beliefs, revealing a confirmation bias that is specifically tied to freely choosing how to sample information. Through two carefully designed studies and detailed analyses, the authors compellingly demonstrate this effect. The novel tasks and models presented here, with all materials freely available, make an important contribution that can benefit other researchers in the field.
By adding a sampling phase between two choices with confidence ratings, these novel tasks allow assessing how an initial choice and confidence influence further information gathering and subsequent choices. The use of eye-tracking and gaze-contingent stimuli display in experiment 2 enables the experimental control needed to …Reviewer #3 (Public Review):
Kaanders et al investigated how information sampling is biased by human agents to support their previous beliefs, revealing a confirmation bias that is specifically tied to freely choosing how to sample information. Through two carefully designed studies and detailed analyses, the authors compellingly demonstrate this effect. The novel tasks and models presented here, with all materials freely available, make an important contribution that can benefit other researchers in the field.
By adding a sampling phase between two choices with confidence ratings, these novel tasks allow assessing how an initial choice and confidence influence further information gathering and subsequent choices. The use of eye-tracking and gaze-contingent stimuli display in experiment 2 enables the experimental control needed to create the fixed sampling conditions that allow disentangling the role of agency in biased information sampling. This demonstrates that unequal information sampling is not sufficient to influence subsequent choices, but that freely deciding how to sample the environment is needed to observe a confirmation bias. This extends previous work showing that confirmation bias was tied to free choices, to show that it is also tied to free information sampling. Finally, the authors develop a novel model that formalises how information gathering is biased to confirm an initial choice, demonstrating that its predictions align with their data, and offering new avenues of exploration of other implications and applications of the model.The conclusions of this report are well supported by the data, but the article could be strengthened by adding some details and considering some further analysis.
The new model is an interesting idea and its impact would be increased by further discussing it within the main article, rather than in the supplementary materials.
Since confidence ratings were also obtained at the second choice, it seems it would be interesting to analyse those to assess whether these are similarly influenced specifically by actively biased information sampling, e.g. allowing people to feel even more confident in the second choice, but also how that interacts with potential changes of mind.
The gaze-contingency and monitoring used in Experiment 2 seems to offer a strong solution to the problem of ensuring that participants are gathering information about both stimuli in a proportion that can be controlled by the experimenter. Yet, as that still allows participants some agency over where to look at, more details about this procedure would help the reader understand how this is done, as would reporting an analysis of the observed looking times for the chosen vs unchosen stimuli in the free vs. fixed sampling conditions (even if in the supplementary materials).
-
