Trading mental effort for confidence in the metacognitive control of value-based decision-making

  1. Douglas G Lee
  2. Jean Daunizeau

  • Curated by eLife

    eLife logo

    Summary: This manuscript addresses a timely subject: the role of cognitive control (or mental effort) in value-based decision making. While there are plenty of models explaining value-based choice, and there is a growing number of computational accounts concerning effort-allocation, little theoretical work has been done to relate the two literatures. This manuscript contributes a novel and interesting step in this direction, by introducing a computational account of meta-control in value-based decision making. According to this account, meta-control can be described as a cost-benefit analysis that weighs the benefits of allocating mental effort against associated costs. The benefits of mental effort pertain to the integration of value-relevant information to form posterior beliefs about option values. Given a small set of parameters, as well as pre-choice value ratings and pre-choice uncertainty ratings as inputs to the model, it can predict relevant decision variables as outputs, such as choice accuracy, choice confidence, choice induced preference changes, response time and subjective effort ratings. The study fits the model to data from a behavioral experiment involving value-based decisions between food items. The resulting behavioral fits reproduce a number of predictions derived from the model. Finally, the article describes how the model relates to established accumulator models of decision-making.

    The (relatively simple) model is impressive in its apparent ability to reproduce qualitative patterns across diverse data including choices, RTs, choice confidence ratings, subjective effort, and choice-induced changes in relative preferences successfully. The model also appears well-motivated, well-reasoned, and well-formulated. While all reviewers agreed that the manuscript is of potential interest, they also all felt that a stronger case needs to be made for the explanatory power of the model, and that the model should be embedded more thoroughly in the existing literature on this topic.

Reviewed by

Read the full article See related articles

Discuss this preprint

Start a discussion What are Sciety discussions?

Listed in

Log in to save this article

Abstract

Why do we sometimes opt for actions or items that we do not value the most? Under current neurocomputational theories, such preference reversals are typically interpreted in terms of errors that arise from the unreliable signaling of value to brain decision systems. But, an alternative explanation is that people may change their mind because they are reassessing the value of alternative options while pondering the decision. So, why do we carefully ponder some decisions, but not others? In this work, we derive a computational model of the metacognitive control of decisions or MCD. In brief, we assume that fast and automatic processes first provide initial (and largely uncertain) representations of options' values, yielding prior estimates of decision difficulty. These uncertain value representations are then refined by deploying cognitive (e.g., attentional, mnesic) resources, the allocation of which is controlled by an effort-confidence tradeoff. Importantly, the anticipated benefit of allocating resources varies in a decision-by-decision manner according to the prior estimate of decision difficulty. The ensuing MCD model predicts response time, subjective feeling of effort, choice confidence, changes of mind, as well as choice-induced preference change and certainty gain. We test these predictions in a systematic manner, using a dedicated behavioral paradigm. Our results provide a quantitative link between mental effort, choice confidence, and preference reversals, which could inform interpretations of related neuroimaging findings.

Article activity feed

  1. Version published to 10.7554/elife.63282 on eLife
  2. eLife

    Reviewer #3:

    Lee and Daunizeau formulate a model of the effects of mental effort on the precision and mode of value representations during value-based decision-making. The model describes how optimal levels of effort can be determined from initial estimates of precision and relative value difference between competing alternatives, accounting for the subjective cost of incremental effort investment, as well as its impact on precision and value differences. This relatively simple model is impressive in its apparent ability to reproduce qualitative patterns across diverse data including choices, RTs, choice confidence ratings, subjective effort, and choice-induced changes in relative preferences successfully. The model also appears well-motivated, well-reasoned, and well-formulated.

    I have two sets of concerns, my first set relates to model fitting and validation. The model appears to do fairly well in predicting aggregate, group-level data, but does it predict subject-level data? Or, does it sometimes make unrealistic predictions when fitting to individual subjects? The Authors should provide evidence of whether it can or cannot describe subject level choices, confidence ratings, subjective effort, etc.

    Also, I think the Authors should do more to demonstrate that their model is an advance on simpler variants. The closest thing to model comparison is an exercise where the authors show that, relative to when their model is fit to random data, their model explains more variance in dependent variables when fit to real data. This exercise uses a straw man as a baseline because almost any model which systematically relates independent variables to dependent variables would explain more variance when fit to real data than to data for which, by definition, independent and dependent variables do not share variance. It would be more useful to know whether (and if so, how much) their model explains data better, than, e.g. a model with where effort only affects precision (beta efficacy), or a model in which effort only impacts value mode (gamma efficacy). Since the Authors pit their model against evidence accumulation models, it would be yet more useful to ask whether their data predicts these diverse data better than a standard evidence accumulation model variants.

    My second set of concerns are regarding the assumed effect of mental effort on the mode of subjective values. First, is it reasonable to assume that variance would increase as a linear function of resource allocation? It seems to me that variance might increase initially, but then each increment of resources would add diminishing variance to the mode since, e.g., new mnesic evidence should tend to follow old mnesic evidence. How sensitive are model predictions to this assumption? What about if each increment of resources added to variance in an exponentially decreasing fashion? Also, what about anchoring biases? Because anchoring biases suggest that we estimate things with reference to other value cues, should we always expect that additional resources increase the expected value difference, or might additional effort actually yield smaller value differences over time? If we relax this assumption, how does this impact model predictions?

  3. eLife

    Reviewer #2:

    The manuscript introduces a computational account of meta-control in value-based decision making. According to this account, meta-control can be described as a cost-benefit analysis that weighs the benefits of allocating mental effort against associated costs. The benefits of mental effort pertain to the integration of value-relevant information to form posterior beliefs about option values. Given a small set of parameters, as well as pre-choice value ratings and pre-choice uncertainty ratings as inputs to the model, it can predict relevant decision variables as outputs, such as choice accuracy, choice confidence, choice induced preference changes, response time and subjective effort ratings. The study fits the model to data from a behavioral experiment involving value-based decisions between food items. The resulting behavioral fits reproduce a number of predictions derived from the model. Finally, the article describes how the model relates to well-established accumulator models, such as the drift diffusion model or the race model.

    Before I get into more detailed comments, I would like to highlight that this work addresses a timely and heavily debated subject, namely the role of cognitive control (or mental effort) in value-based decision making (see Shenhav et al., 2020). While there are plenty of models explaining value-based choice, and there is a growing number of computational accounts concerning effort-allocation, little theoretical work has been done to relate the two literatures (but see Major Comment 1). This work contributes a novel and interesting step in this direction. Moreover, I had the impression that the presented model can account for a broad range of behavioral phenomena and that the authors did a commendable amount of work to validate the model (but see Major Comments 2 and 3). The manuscript is also well written in that it seems accessible to a broad audience, including non-technical readers. However, while I remain curious about what the other reviewers have to say, the manuscript misses to address a few issues that I elaborate below.

    Major Comments:

    1. Model Comparison(s): While the manuscript compares the presented computational approach to existing accumulator models, it could situate itself better in the existing literature, ideally in the form of formal model comparisons. For instance, as someone less familiar with choice-induced preference changes in value-based decision making, I wonder how the model compares to existing computational work on this matter, e.g. the models described in Izuma & Murayama (2013) or the efficient coding account of Polanía, Woodford, & Ruff (2019). I do understand that the presented model can account for some phenomena that the other models cannot account for, at least without auxiliary assumptions (e.g. subjective effort ratings), but the interested reader might want to know how well the presented model can explain established decision-related variables, such as decision confidence, choice accuracy or choice-induced preference changes compared to existing models, by having them contrasted in a formal manner. Finally, it would seem fair to compare the presented account to emerging, more mechanistically explicit accounts of meta-control in value-based decision making (e.g. Callaway, Rangel & Griffiths, 2020; Jang, Sharma, & Drugowitsch, 2020). As these approaches are still in preprint, it may not be necessary to relate them in a formal model comparison. However, the manuscript might benefit from discussing how these approaches differ from the presented model in the text.

    2. Fitting Procedure: This comment concerns the validation of the described model based on its fits to behavioral data. If I understand correctly, the authors first fit the model to each participant while "[a]ll five MCD dependent variables were [...] fitted concurrently with a single set of subject-specific parameters" and then evaluate whether model fits match the predicted qualitative relationship between experimental variables (e.g. pre-choice value ratings and pre-choice confidence ratings) and dependent variables (e.g. choice accuracy). I'm happy to be convinced otherwise, but it appears that the model's predictions could be tested in a more stringent manner. That is, it doesn't appear compelling to me that the model, once fitted, matches the behavior of participants -- please note that this is not to diminish the value of the results; I still think that these results are valuable to include in the manuscript. Instead, rather than fitting the model to all dependent variables at once, it would be more compelling to fit the model to a subset of established decision-related variables (e.g. accuracy, choice confidence, choice induced preference changes) and then evaluate how the fitted model can predict out-of-sample variables related to effort allocation (e.g. response time and subjective effort ratings). Again, I am happy to be convinced otherwise but the latter would seem like a much more stringent test of the model, and may serve to highlight its value for linking variables related to value-based decision making to variables related to meta-control.

    3. Parameter Recoverability: Given that many of the results rely on model fits to human participants, it would seem appropriate to include an analysis of parameter recoverability. That is how well can the fitting procedure recover model parameters from data generated by the model? I apologize if I missed this, but the manuscript doesn't appear to report this kind of analysis.

    References:

    Callaway, F., Rangel, A., & Griffiths, T. L. (2020). Fixation patterns in simple choice are consistent with optimal use of cognitive resources. PsyArXiv: https://doi.org/10.31234/osf.io/57v6k

    Izuma, K., & Murayama, K. (2013). Choice-induced preference change in the free-choice paradigm: a critical methodological review. Frontiers in psychology, 4, 41.

    Jang, A. I., Sharma, R., & Drugowitsch, J. (2020). Optimal policy for attention-modulated decisions explains human fixation behavior. bioRxiv: 2020.2008.2004.237057.

    Polania, R., Woodford, M., & Ruff, C. C. (2019). Efficient coding of subjective value. Nature neuroscience, 22(1), 134-142.

    Shenhav, A., Musslick, S., Botvinick, M. M., & Cohen, J. D. (2020, June 16). Misdirected vigor: Differentiating the control of value from the value of control. PsyArXiv: https://doi.org/10.31234/osf.io/5bhwe

  4. eLife

    Reviewer #1:

    The authors report a model about the confidence-effort tradeoff; explaining how subjects invest effort depending on how confident they want to be in their decision (and how costly this is). They fit their model to behavioural data and report qualitative similarities between model and data.

    I find this an interesting model, with interesting links between timely topics of interest, such as confidence, effort, and cost optimisation. But I have several requests for clarification.

    Major Comments:

    Line 274: Without loss of generality: what does it mean here? I guess that with a different cost function, not all conclusions remain the same?

    The model assumes that it is "rewarding" to choose the correct (highest-value) option (B = R*P). But is this realistic? If the two options have approx the same value, then R should be small (it doesn't matter which one you choose); if the options have different values, it is important to choose the correct one. Of course, the probability P_c continuously differentiates between the two options, but that is not the same as the reward. Can the predictions generalise toward a more general R that depends on value difference?

    In Figure 2, I guess that the important quantity to decide is a standardised delta-mu (similar to d' in signal detection theory). It might be useful to also plot that (essentially combining the current two plots). Or alternatively, plot P_c(z), which relates more directly to the theory.

    The section Probabilistic model fit is unclear. Are the MCD variables y the 5 variables mentioned above? Do different y's share the same alpha, beta, gamma? Are different transformation parameters a and b fitted for each y? Is estimation done per subject? It is mentioned that VBA is used, but what distribution is approximated exactly using VBA? Is it a mean-field approximation, optimised with gradient descent? Is the goal function a posterior across the 5 parameters? It would also be good then to have an intuition on the estimated model parameters (e.g., their standard error or Bayesian equivalent). Is there an estimate of model fit (in addition to checking qualitative predictions)? Figure S3 is a good start (and I think it is worth putting in the main MS), but it would be nice, for example, to see model comparisons where one or more parameters are restricted.

    Figure 4, 5, 6 should be better annotated. I have a hard time trying to fill in what is plotted exactly (eg, scale of the color bar). Why are the data grouped in percentiles? Also in Figure 4 legend, I guess that "beta" is not used as the MCD model parameter? Please avoid overloading definitions.

    Figure 7: It seems that "spreading" of alternatives occurs in the model only for alternatives that are initially close together? Is this consistent with their discussion around equation (14)? (I may be overlooking something; if so, consider making this more explicit.)

    I find it a really interesting feature of the model that it can explain spreading of alternatives from a statistical perspective. So I think it's worth commenting on it in the Discussion. For example, does the current model capture trends in the literature? To what extent is the effect (also in empirical data) dependent on initial value differences?

  5. eLife

    Summary: This manuscript addresses a timely subject: the role of cognitive control (or mental effort) in value-based decision making. While there are plenty of models explaining value-based choice, and there is a growing number of computational accounts concerning effort-allocation, little theoretical work has been done to relate the two literatures. This manuscript contributes a novel and interesting step in this direction, by introducing a computational account of meta-control in value-based decision making. According to this account, meta-control can be described as a cost-benefit analysis that weighs the benefits of allocating mental effort against associated costs. The benefits of mental effort pertain to the integration of value-relevant information to form posterior beliefs about option values. Given a small set of parameters, as well as pre-choice value ratings and pre-choice uncertainty ratings as inputs to the model, it can predict relevant decision variables as outputs, such as choice accuracy, choice confidence, choice induced preference changes, response time and subjective effort ratings. The study fits the model to data from a behavioral experiment involving value-based decisions between food items. The resulting behavioral fits reproduce a number of predictions derived from the model. Finally, the article describes how the model relates to established accumulator models of decision-making.

    The (relatively simple) model is impressive in its apparent ability to reproduce qualitative patterns across diverse data including choices, RTs, choice confidence ratings, subjective effort, and choice-induced changes in relative preferences successfully. The model also appears well-motivated, well-reasoned, and well-formulated. While all reviewers agreed that the manuscript is of potential interest, they also all felt that a stronger case needs to be made for the explanatory power of the model, and that the model should be embedded more thoroughly in the existing literature on this topic.

  7. Version published to 10.1101/837054 on bioRxiv