Benchmarking the accuracy of structure‐based binding affinity predictors on Spike–ACE2 deep mutational interaction set

  1. Burcu Ozden
  2. Eda Şamiloğlu
  3. Atakan Özsan
  4. Mehmet Erguven
  5. Can Yükrük
  6. Mehdi Koşaca
  7. Melis Oktayoğlu
  8. Muratcan Menteş
  9. Nazmiye Arslan
  10. Gökhan Karakülah
  11. Ayşe Berçin Barlas
  12. Büşra Savaş
  13. Ezgi Karaca

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

Since the start of COVID‐19 pandemic, a huge effort has been devoted to understanding the Spike (SARS‐CoV‐2)–ACE2 recognition mechanism. To this end, two deep mutational scanning studies traced the impact of all possible mutations across receptor binding domain (RBD) of Spike and catalytic domain of human ACE2. By concentrating on the interface mutations of these experimental data, we benchmarked six commonly used structure‐based binding affinity predictors (FoldX, EvoEF1, MutaBind2, SSIPe, HADDOCK, and UEP). These predictors were selected based on their user‐friendliness, accessibility, and speed. As a result of our benchmarking efforts, we observed that none of the methods could generate a meaningful correlation with the experimental binding data. The best correlation is achieved by FoldX ( R  = −0.51). When we simplified the prediction problem to a binary classification, that is, whether a mutation is enriching or depleting the binding, we showed that the highest accuracy is achieved by FoldX with a 64% success rate. Surprisingly, on this set, simple energetic scoring functions performed significantly better than the ones using extra evolutionary‐based terms, as in Mutabind and SSIPe. Furthermore, we demonstrated that recent AI approaches, mmCSM‐PPI and TopNetTree, yielded comparable performances to the force field‐based techniques. These observations suggest plenty of room to improve the binding affinity predictors in guessing the variant‐induced binding profile changes of a host–pathogen system, such as Spike–ACE2. To aid such improvements we provide our benchmarking data at https://github.com/CSB-KaracaLab/RBD-ACE2-MutBench with the option to visualize our mutant models at https://rbd-ace2-mutbench.github.io/ .

Article activity feed

  1. Version published to 10.1002/prot.26645
  2. Version published to 10.1101/2022.04.18.488633v4 on bioRxiv
  3. Version published to 10.1101/2022.04.18.488633v3 on bioRxiv
  4. Version published to 10.1101/2022.04.18.488633v2 on bioRxiv
  5. ScreenIT

    SciScore for 10.1101/2022.04.18.488633: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    NIH rigor criteria are not applicable to paper type.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    The heatmaps in Figure 1B were generated with pandas, Numpy and seaborn libraries of Python 3.8 (29–34).
    Numpy
    suggested: (NumPy, RRID:SCR_008633)
    Python
    suggested: (IPython, RRID:SCR_001658)
    MutaBind2 and SSIPe: use FoldX and EvoEF1, respectively, to explicitly model the desired mutation.
    FoldX
    suggested: (FoldX, RRID:SCR_008522)
    Next to these, for scoring, MutaBind2 utilizes other force field and contact-based terms, together with a metric measuring the evolutionary conservation of the mutated site (36).
    MutaBind2
    suggested: None
    Success rate and metric evaluations were performed in Python 3.8.5 with Pandas, Numpy, seaborn, and Matplotlib libraries (29–34, 45).
    Matplotlib
    suggested: (MatPlotLib, RRID:SCR_008624)

    Results from OddPub: Thank you for sharing your code and data.

    Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    Results from scite Reference Check: We found no unreliable references.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  6. Version published to 10.1101/2022.04.18.488633v1 on bioRxiv