A statistical framework for assessing pharmacological response and biomarkers using uncertainty estimates
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (eLife)
Abstract
Drug high-throughput screenings across large molecular-characterised cancer cell line panels enable the discovery of biomarkers, and thereby, cancer precision medicine. The ability to experimentally generate drug response data has accelerated. However, this data is typically quantified by a summary statistic from a best-fit dose response curve, whilst neglecting the uncertainty of the curve fit and the potential variability in the raw readouts. Here, we model the experimental variance using Gaussian Processes, and subsequently, leverage this uncertainty for identifying associated biomarkers with a new statistical framework based on Bayesian testing. Applied to the Genomics of Drug Sensitivity in Cancer, in vitro screening data on 265 compounds across 1,074 cell lines, our uncertainty models identified 24 clinically established drug response biomarkers, and in addition provided evidence for 6 novel biomarkers. We validated our uncertainty estimates with an additional drug screen of 26 drugs, 10 cell lines with 8 to 9 replicates. Our method is applicable to drug high-throughput screens without replicates, and enables robust biomarker discovery for new cancer therapies.
Article activity feed
-
###Reviewer #2 In this manuscript, the authors applied Gaussian Process regression to drug response data and attempted to utilize the estimates of uncertainty from these regression to improve on drug response curve fitting and biomarker discovery. Their approach and application case is an interesting one that deserves further investment and attention. However, I have substantive concerns with the current manuscript draft and would recommend to the authors that these concerns be addressed.
- Figure 3 and the accompanying text section of the main document seems to be focused on characterizing estimation uncertainty, which appears to simply be the between-sample dispersion of the dose-response curve (or summary statistics thereof) from replicate runs. The main conclusion seems to be that drug compounds with partial responders are the ones …
###Reviewer #2 In this manuscript, the authors applied Gaussian Process regression to drug response data and attempted to utilize the estimates of uncertainty from these regression to improve on drug response curve fitting and biomarker discovery. Their approach and application case is an interesting one that deserves further investment and attention. However, I have substantive concerns with the current manuscript draft and would recommend to the authors that these concerns be addressed.
- Figure 3 and the accompanying text section of the main document seems to be focused on characterizing estimation uncertainty, which appears to simply be the between-sample dispersion of the dose-response curve (or summary statistics thereof) from replicate runs. The main conclusion seems to be that drug compounds with partial responders are the ones with the greatest between-sample dispersions.
What is missing from this Figure and accompanying text is a comparison of these results with analogous ones for the observation uncertainty to help readers understand why one approach may be preferred over the other.
Figure 5A compares the posterior probability from the Bayesian test (presumably accounting for estimation uncertainty) against the q-value from an ANOVA test. The q-value should be the False Discovery Rate, which controls for the proportion of false positives. This does not seem to be directly comparable to a posterior probability. The authors should clarify why a comparison of proportion to posterior probability is reasonable.
The authors do not appear to have demonstrated how estimation uncertainty can improve on drug response curve fitting or biomarker discovery?
For the former, the fitted curves using standard approaches appear similar to those fitted using GP regression, as the authors seemed to have focused on those curves where the two approaches are concordant and as the IC50 value differences appear minimal for those cases where IC50 is within the tested concentration range. The greatest differences are seen for those cases where IC50 values are outside the tested concentration ranges, but these cases were not in focus in the text. In addition, for these cases, it is unclear if relying on curve fits from GP regression makes sense because they are also the cases with the highest estimation uncertainty.
For the latter, it appears that every significant biomarker identified using Bayesian posterior probability is also significant by ANOVA (using a standard q-value < 0.05 cutoff).
-
###Reviewer #1 The authors propose two related (though distinct) methods for the improvement of pharmacological screening analysis and related biomarker analyses. The first is a Gaussian process (GP) approach to dose-response curve fitting for the estimation of IC50, AUC, and related quantities. The goal of this method is to improve point and uncertainty estimates of these quantities through more flexible functional specification and outlier-robust error modeling. The second method is a hierarchical Bayesian approach to biomarker association analysis. This incorporates uncertainty estimates produced by the GP modeling with the aim of providing more sensitive association analyses with fewer false positives.
The combination of methods presented has some potential. Flexible modeling of dose-response relationships and better estimation of …
###Reviewer #1 The authors propose two related (though distinct) methods for the improvement of pharmacological screening analysis and related biomarker analyses. The first is a Gaussian process (GP) approach to dose-response curve fitting for the estimation of IC50, AUC, and related quantities. The goal of this method is to improve point and uncertainty estimates of these quantities through more flexible functional specification and outlier-robust error modeling. The second method is a hierarchical Bayesian approach to biomarker association analysis. This incorporates uncertainty estimates produced by the GP modeling with the aim of providing more sensitive association analyses with fewer false positives.
The combination of methods presented has some potential. Flexible modeling of dose-response relationships and better estimation of uncertainty are interesting axes to wring more information out of large-scale screening datasets. There are a few areas to shore up in the paper to increase confidence in the empirical results and generalizability of the methods.
There are a number of fixed parameters in the proposed methods, and the calibration procedure used to set these is unclear to me. For the GP models, there are a set of noise parameters for Beta mixture and the length scales and variance parameter for the kernel. I'm not sure how one would generalize the GP methods to other screening datasets as a result of this ambiguity (e.g., how would one determine appropriate noise parameters?). For the hierarchical Bayesian biomarker association model, we have prior scale parameters related to both the effect size and variance parameters. The number of researcher degrees of freedom introduced by these tuned parameters also raises some concerns about the sensitivity of empirical results (e.g., 24 clinically established biomarkers and 6 novel) to these choices. It's not clear if we're seeing a corner case or a robust result. I think the work would benefit from both sensitivity analyses with respect to tuned parameters and guidance on or methods for their estimation. The latter is particularly important if other researchers hope to employ these methods in a different context.
The proposed hierarchical Bayesian approach to biomarker association analysis is a reasonable start, but it was unclear to me whether changes in performance stemmed from correcting misspecification in original ANOVA or the use of uncertainty estimates. I suggest comparing results to a heteroskedasticity-robust estimator (e.g., HC3, see Long and Ervin, 2000), which would be valid under the stated model without the requirement for explicit uncertainty estimates or priors. The transformations and tuning applied to uncertainty estimates in this context also make generalization of the approach challenging. The need for the c (power) parameter suggests a potential misspecification or miscalibration at some point in the modeling chain. It would be useful to understand this misspecification better, particularly for researchers hoping to extend or reuse these methods.
The GP method provides reasonable estimates of uncertainty, but it would be useful to see them compared to those from the sigmoid model (e.g., from the delta method). It wasn't clear to me how much of the difference in results is coming from incorporation of uncertainty estimates as opposed to changes in the point estimates.
The handling of cases with IC50 beyond the maximum observed dose (extrapolating to 10x the maximum concentration) provided a reasonable starting point, but a few subtleties in the handling of corner cases remain unaddressed (e.g., GPs allow positive slope at right edge of range). It would be useful to provide a more general, systematic procedure to address these. Imposing monotonicity may not be the best path, but additional guidance for researchers applying these methods in other contexts would help.
-
##Preprint Review
This preprint was reviewed using eLife’s Preprint Review service, which provides public peer reviews of manuscripts posted on bioRxiv for the benefit of the authors, readers, potential readers, and others interested in our assessment of the work. This review applies only to version 3 of the manuscript.
###Summary
This manuscript presents two statistical approaches to evaluating drug effect measurements and associations between biomarkers, for dose curve data. Measurements of these kinds are made in many contexts, and frequently reported without accounting well for measurement uncertainties. A statistical framework of this kind will be widely useful and should be frequently applied.
-
-
-