Value Certainty in Drift-Diffusion Models of Preferential Choice

  1. Douglas Lee
  2. Marius Usher

  • Curated by eLife

    eLife logo

    Summary: This study investigates how uncertainty about the values of choice alternatives affects decision-making from the perspective of drift-diffusion modeling. Both reviewers agree that this is an interesting question. The authors propose different candidate models for how uncertainty might affect the drift rate or the diffusion variance, and test these candidates on four food-preference datasets. The authors report that the best model is one in which the drift rate scales with the value of the options normalized by their respective uncertainties.

    Despite the relevance of the research question, both reviewers have found the contribution of the findings to existing knowledge to be not sufficiently strong and clear. Several empirical observations reported in the study are already well known, and several of the alternative models are known to be "strawmen" for researchers in value-based decision-making and drift-diffusion modeling. In particular, the reviewers have noted that is not surprising that a lower certainty alone cannot correspond to higher diffusion noise in a drift-diffusion model, and can thus be captured by a lower drift. They agreed, and further amplified in the consultation session amongst reviewers, that the precise computational way by which this drift modulation is implemented would need to be investigated much further. Furthermore, to increase the strength of the conclusions, the authors should explore in more detail the different classes of DDMs, and the ways in which value certainty could affect other parameters of the model than the ones considered in the manuscript.

Reviewed by

Read the full article See related articles

Discuss this preprint

Start a discussion What are Sciety discussions?

Listed in

Log in to save this article

Abstract

The drift-diffusion model (DDM) is widely used and broadly accepted for its ability to account for binary choices (in both the perceptual and preferential domains) and response times (RT), as a function of the stimulus or the choice alternative (or option) values. The DDM is built on an evidence accumulation-to-bound concept, where, in the value domain, a decision maker repeatedly samples the mental representations of the values of the available options until satisfied that there is enough evidence (or support) in favor of one option over the other. As the signals that drive the evidence are derived from value estimates that are not known with certainty, repeated sequential samples are necessary to average out noise. The classic DDM does not allow for different options to have different levels of precision in their value representations. However, recent studies have shown that decision makers often report levels of certainty regarding value estimates that vary across choice options. There is therefore a need to extend the DDM to include an option-specific value certainty component. We present several such DDM extensions and validate them against empirical data from four previous studies. The data support best a DDM version in which the drift of the accumulation is based on a sort of signal-to-noise ratio of value for each option (rather than a mere accumulation of samples from the corresponding value distributions). This DDM variant accounts for the impact of value certainty on both choice consistency and response time present in the empirical data.

Article activity feed

  1. eLife

    Author Response

    Response to reviews:

    We appreciate the relevant comments sent to us in this review. We have already revised the paper and we addressed those points in our revised manuscript. There is a particular point, which we had not explained in enough detail in our original version of the paper, and which we believe has led the reviewers to not appreciate a central aspect of our study. We wish to clarify this below:

    The reviewers stated that the idea proposed in our study that the "drift rate corresponds to signal-to-noise ratio" is a quite accepted one in DDM research, which typically assumes that the "within-trial noise" magnitude is fixed (and does not vary with condition), while drifts do. From this, it also followed that one the models we examined (and rejected; our model 2) appears to be a 'strawman', which one would NOT seriously consider.

    REPLY:

    This statement could be correct with regard to the DDM framework, within the domain of perceptual choice. However, we focused here on the DDM extension to value-based decisions, and we believe that the statement above is no longer accurate.

    1. Noise magnitude and within-trial sampling variability in value-based DDM

    Whereas perceptual choices are usually brief (often between .5 - 1 sec) and the stimuli they present are often static (lines or strings of letters in a lexical decision), value-based decisions take longer (typically around 2-5 sec), and the values for which they accumulate evidence are not “given” but rather need to be generated (sampled) during the decision itself. While all versions of the DDM include accumulation noise, the difference pointed out above has made its application to perceptual decisions assume that the "accumulation" term is constant and does not vary with task difficulty (this was also motivated by the attempt to minimize model parameters, so it was thought one could keep this parameter fixed). While this practice has been criticized (Donkin, Brown & Heatcote, 2009), the fact that the tasks involve short and roughly static stimuli (so the accumulation noise may be small compared with noise that appears between trials, when the same stimulus is presented again) has led most researchers to either assume the accumulation noise is fixed or to neglect it altogether (in favor of between trial noise; LBA-model).

    The first application of a sequential sampling model to value based decisions was the decision-field theory (DFT) model (e.g., Busemeyer & Diederich, 2002; Hotaling & Busemeyer, 2012). In this model, accumulation is driven by attentional switches between dimensions that are relevant to distinguish the stimuli, resulting in an explicitly noisy accumulation. More recent application of the DDM to value-based decisions (e.g., Krajbich et al, 2010; Tajima et al., 2016) are consistent with this idea. For example, as described by Tajima et al, the values of the alternatives are not "known" by the subject (even if the alternatives are in full view), but rather they are sampled from a distribution whose width corresponds to their previous experience or knowledge of the alternatives. Thus, in this framework, the within-trial accumulation becomes an intrinsically noisy process. Moreover, as mathematically proved in the DFT model, the within trial accumulation noise is determined by the variance of the sampled values. As long as it was assumed that the distributions of rewards associated with each alternative had equal width, it was possible to assume that the noise term was constant. Since we now know that alternatives vary not only in their attractiveness rating, but also on their certainty about such rating, the most natural assumption is that subjects accumulate value “evidence” by sampling from Gaussian distributions whose means correspond to the options’ value ratings and whose variances correspond to the options’ value uncertainties. This leads directly to our Model 2.

    We understand that this model cannot account for the observed data, and in this sense it is not a true contender. However, given the theoretical rationale above, we believe that showing this explicitly should have (at least) a didactical value for the readers of this literature, who want to understand how certainty should be addressed. Obviously, our results support an alternative model in which the drift of the accumulation process (and not the noise) is affected by the certainty of the alternatives. While this is consistent with what the reviewers believe to be expected, in our reading of the value-based decision literature, we did not find any model in which this was explicitly stated or tested. We believe that these results will motivate further investigation into the mechanism that generates this "normalization" (we aim to discuss a few options in our Discussion section).

    1. More detailed DDM explorations for the certainty effect

    We agree that a more detailed investigation of variants of our Model 4 would be informative. Both reviewers have provided very helpful and relevant suggestions, which we have addressed in our revised manuscript.

    For example, we examined a variant of Model 4 in which the drift decrement with uncertainty is non-linear (we introduced an exponent to characterize this). The model fitting results show that, indeed, this model flexibility is beneficial, resulting in better fits (including the flexibility costs). While the average exponent is close to 1 (the average across the group is .85), there is significant variability between subjects resulting in improved data fits. We also carried out a median-split analysis based on the certainty of the options, in which we allowed both the drift and the accumulation noise to vary with certainty. The results were consistent with our previous conclusions, showing that certainty affects the drift but not the accumulation variability. While this may go beyond the scope of the present paper, we will discuss potential mechanisms that might cause these results.

    References:

    Busemeyer, J. R., & Diederich, A. (2002). Survey of decision field theory. Mathematical Social Sciences, 43(3), 345-370.

    Donkin, C., Brown, S. D., & Heathcote, A. (2009). The overconstraint of response time models: Rethinking the scaling problem. Psychonomic Bulletin & Review, 16(6), 1129-1135.

    Hotaling, J. M., & Busemeyer, J. R. (2012). DFT-D: A cognitive-dynamical model of dynamic decision making. Synthese, 189(1), 67-80.

    Krajbich, I., Armel, C., & Rangel, A. (2010). Visual fixations and the computation and comparison of value in simple choice. Nature neuroscience, 13(10), 1292-1298.

    Tajima, S., Drugowitsch, J., & Pouget, A. (2016). Optimal policy for value-based decision-making. Nature communications, 7(1), 1-12.

  2. eLife

    Reviewer #2:

    In this manuscript, Lee and Usher study choices between two options, and model how such choices are affected by the certainty with which the decision-maker evaluates the two options. They insist that this value certainty should be incorporated in current models, and compare ways to do so within the framework of the drift-diffusion model (DDM).

    My main concern is that I find the main contribution a bit light. Mathematically, we know that in a DDM higher noise leads to shorter RTs. Empirically, we already know that options rated with low certainty lead to longer RTs (e.g. as demonstrated by the first author in Lee & Coricelli, 2020). So it is not surprising that low certainty cannot correspond to higher noise in a DDM, and might be captured by a lower drift instead. Then, the specific way it can be done deserves to be investigated, but the authors should explore in more details the different classes of models, and the ways in which value certainty could affect other parameters of the model as well.

    Suggestions:

    I would suggest presenting in the introduction more details about how DDM is currently used in studies of value based decisions, to better explain the context of the present work and highlight the specific contribution of the study.

    The authors consider a number of models in the discussion (effects of uncertainty on the bounds, balance of evidence, collapsing bounds, etc.) but do not give the full details of these models. I would suggest including these models in the analyses presented in the result section. Maybe the authors could capitalize on the amount of data they have to do some model fitting, to estimate how the parameters of the DDM would change with value certainty. Parameters of interest are the drift and the drift variability (in the extended version of the DDM) but the authors could also explore the bounds and the variability in the starting point. A basic approach would be to split the data based on value certainty: using a median-split for both options, they could fit separately the choices between 2 options rated with high certainty, and the choices between 2 options rated with low certainty, etc. A more involved approach would be to estimate the effect of value certainty on the parameters in a single analysis across all the data (e.g. using a hierarchical ddm).

    Minor points:

    The motivation for model 5, which includes an additional component for accumulating certainty, should be more detailed. This approach is not standard, and would deserve more details and some references to prior work offering the same approach, if it exists.

    A figure would be helpful to present the typical experimental paradigm, and including the notations of the variables.

    In Figure 2, the variable C1 and C2 are not properly defined.

  3. eLife

    Reviewer #1:

    This article investigates how uncertainty about the value of alternatives affects the decision process through the lens of the drift diffusion model. The article proposes several models for how uncertainty might affect the drift rates or diffusion variance, and tests those models on four different food-choice datasets. The authors conclude that the best model is one in which the drift rate depends on the values of the options divided by their degree of uncertainty.

    I think the article is pursuing an interesting question. The core set of results are perhaps not as surprising or as puzzling to a DDM audience as the introduction might have you believe, but from there the paper does a nice job of exploring different ways in which uncertainty might affect the choice process. This seems like a good set of models to consider, as they cover the obvious ways in which one might consider incorporating uncertainty into the DDM, and each one, except for the favored Model 4, has a clear inability to capture a facet of the data.

    1. I could quibble about why the authors don't explore more variants of the favored Model 4, for example ones where the values are divided by non-linear functions of the uncertainty measure (e.g. squared or square root)? The results in Figure 4 are not a slam dunk for Model 4, as the effect of dC seems to outweigh C, while in the data it is the opposite. I don't think this is critical, but the authors might try an extra exponent parameter on uncertainty in Model 4. At minimum, the authors should discuss how they might modify Model 4 to better match the data.

    2. As I alluded to above, I think the article somewhat mischaracterizes the DDM by saying that "the most straightforward way to include option-specific noise in the preferential DDM - by assuming that noise increases with value uncertainty - leads to the wrong qualitative predictions..." "Most straightforward" is subjective. The standard diffusion model sets the diffusion noise variance to a constant, and so no, adjusting the noise is not "straightforward"; in many DDM software packages it is not even an option. Instead the effect of uncertainty would show up in the drift rate (or boundaries), as it does here. So, I would urge the authors to temper their claims in the introduction and discussion about what the "straightforward" model would be. Many researchers who use the DDM think about the drift rate as a signal-to-noise ratio, and for them Model 4 would have been the straightforward model.

    3. This isn't to say that what the article does isn't interesting or important. A standard DDM analysis would just fit different drift-rate and boundary parameters to high and low uncertainty conditions and then report the differences. This article takes a more elegant approach by explicitly modeling uncertainty in the DDM components. This is why I would urge the authors to do a bit more with that aspect of the paper, to try to better understand how uncertainty impacts the drift rates.

    4. On Page 16 - the authors write "in line with the best fit parameters". What exactly do they mean here? Did they use the best-fitting parameters or not? Could the authors add a table to the supplements with the average best-fitting parameters for each model, for each dataset? That would greatly help in understanding the results.

    5. Figure 4 - how were the experimental data and model simulations combined to generate these figures? For the data, was this one big mixed-effects regression including all datasets? How did the authors handle the random effects in this case, given the multiple datasets? The simulations are also vaguely described. How "similar" were the input values to the data; how exactly were these input values generated? Again, how were the simulations from different subjects/studies combined to generate a single plot per model? It would be useful, though not strictly necessary, to see the basic behavioral results broken down by study (in the supplements). It is unclear how consistent the patterns in Figure 2/4 are across the studies.

  4. eLife

    Summary: This study investigates how uncertainty about the values of choice alternatives affects decision-making from the perspective of drift-diffusion modeling. Both reviewers agree that this is an interesting question. The authors propose different candidate models for how uncertainty might affect the drift rate or the diffusion variance, and test these candidates on four food-preference datasets. The authors report that the best model is one in which the drift rate scales with the value of the options normalized by their respective uncertainties.

    Despite the relevance of the research question, both reviewers have found the contribution of the findings to existing knowledge to be not sufficiently strong and clear. Several empirical observations reported in the study are already well known, and several of the alternative models are known to be "strawmen" for researchers in value-based decision-making and drift-diffusion modeling. In particular, the reviewers have noted that is not surprising that a lower certainty alone cannot correspond to higher diffusion noise in a drift-diffusion model, and can thus be captured by a lower drift. They agreed, and further amplified in the consultation session amongst reviewers, that the precise computational way by which this drift modulation is implemented would need to be investigated much further. Furthermore, to increase the strength of the conclusions, the authors should explore in more detail the different classes of DDMs, and the ways in which value certainty could affect other parameters of the model than the ones considered in the manuscript.

  5. Version published to 10.1101/2020.08.22.262725 on bioRxiv