Exploring the accuracy of ab initio prediction methods for viral pseudoknotted RNA structures

  1. Vasco Medeiros
  2. Jennifer M. Pearl
  3. Mia Carboni
  4. Ece Er
  5. Stamatia Zafeiri

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

The prediction of tertiary RNA structures is significant to the field of medicine (e.g. mRNA vaccines, genome editing), and the exploration of viral transcripts. Though many RNA folding software exist, few studies have condensed their locus of attention solely to viral pseudoknotted RNA. These regulatory pseudoknots play a role in genome replication, gene expression, and protein synthesis. This study explores five RNA folding engines that compute either the minimum free energy (MFE) or the maximum expected accuracy (MEA). These folding engines were tested against 26 experimentally derived short pseudoknotted sequences (20-150nt) using metrics that are commonly applied to software prediction accuracy (e.g. F 1 scoring, PPV). This paper reports higher accuracy RNA prediction engines, such as pKiss, when compared to previous iterations of the software, and when compared to older folding engines. They show that MEA folding software does not always outperform MFE folding software in prediction accuracy when assessed with metrics such as percent error, sensitivity, PPV, and F 1 scoring when applied to viral pseudoknotted RNA. Moreover, the results suggest that thermodynamic model parameters will not ensure accuracy if auxiliary parameters such as Mg 2+ binding, dangling end options, and H-type penalties are not applied. The observations reported in this paper highlight the quality between different ab initio prediction methods while enforcing the idea that a better understanding of intracellular thermodynamics is necessary for a more efficacious screening of RNAs.

Importance

The importance of accurately predicting RNA structures cannot be overstated, particularly in the context of viral biology and the development of therapeutic interventions such as mRNA vaccines and genome editing. Our study addresses the gap in the existing literature by concentrating solely on viral pseudoknotted RNA, which plays a crucial role in viral replication, gene expression, and protein synthesis. Our study sheds light on the debate surrounding minimum free energy (MFE) versus maximum expected accuracy (MEA) models in RNA folding predictions. Contrary to existing beliefs, we found that MEA models do not consistently outperform MFE models, especially in the context of viral pseudoknotted RNAs. Our research contributes to advancing the field of computational biology by providing insights into the efficacy of different prediction methods and emphasizing the need for a deeper understanding of intracellular thermodynamics to improve RNA structure predictions.

Article activity feed

  1. PREreview

    This Zenodo record is a permanently preserved version of a PREreview. You can view the complete PREreview at https://prereview.org/reviews/12693185.

    This review is the result of a virtual, collaborative live review discussion organized and hosted by PREreview and JMIR Publications on June 20, 2024. The discussion was joined by 11 people: 2 facilitators, 2 members of the JMIR Publications team, 1 author, and 6 live review participants including two who agreed to be named: Mike Chang and Heba Abdullah Mohammed Ali. The authors of this review have dedicated additional asynchronous time over the course of two weeks to help compose this final report using the notes from the Live Review. We thank all participants who contributed to the discussion and made it possible for us to provide feedback on this preprint.

    Summary

    The study examines the performance of 5 RNA folding engines in predicting complex viral pseudoknotted RNA structures. This research fills a critical gap in the field by comparing the efficiency of minimal free energy (MFE) and maximum expected accuracy (MEA) using a curated dataset of 26 viral RNA sequences with known secondary structures. Contrary to prevailing assumptions favouring MEA models, their findings reveal that pKiss, an MFE folding engine outperforms Vsfold 5 in terms of sensitivity, positive predictive value (PPV), and F1 scores, while laying emphasis on the importance of PPV and sensitivity parameters in understanding and determining the superior accuracy of pKiss to predict correct base pairs and  minimize incorrect predictions. The authors also point out that the said engine still needed additional data to achieve high accuracy as well as a better understanding of thermodynamics at the intracellular level.

    The statistical analyses used to evaluate the results were two-way ANOVA and Tukey's multiple comparisons test, which provided robust insights to performance differences among the tested engines. The research integrates bioinformatics with statistics and advanced data science methodologies to promote our understanding of computational RNA biology. The study provides important insights into the relative advantages and disadvantages of both approaches in predicting pseudoknotted RNA structures by contrasting minimal free energy (MFE) models and maximum expected accuracy (MEA) models. It also highlights avenues for future research to focus on the development of more sophisticated energy models and MFE engines, like pKiss to enhance prediction capabilities most especially in the context of viral replication and gene regulation, which may lead to a better understanding of the functional roles of pseudoknotted RNA structures. Overall, this research contributes significantly to the field of computational and molecular biology.

    Below we list major and minor concerns that were discussed by participants of the Live Review and, where possible, we provide suggestions on how to address those issues.

    List of major concerns and feedback

    • It would be helpful to provide more context on why Percent Error was chosen as the primary metric for evaluating different engines. Considering alternatives like Mean Absolute Error (MAE) and Mean Squared Error (MSE) could enhance the analysis. For instance, MAE is robust against outliers, making it a valuable metric, especially when outlier removal is part of the process. Although MAE is less sensitive to extreme values, it can offer a useful qualitative check on the models. On the other hand, MSE's sensitivity to outliers can be advantageous when the spread of the forecast is important. Including these metrics could provide a more comprehensive evaluation.

    • The authors have conducted a comprehensive and insightful study, revealing important differences in prediction accuracy between Vsfold 5 and pKiss. One area that could further enhance the manuscript is the exploration of how auxiliary parameters (e.g., Mg2+ binding, dangling end options, H-type penalties) are managed across the various RNA folding engines utilized. For example, Vsfold 5, although being an MEA model, may encounter challenges if its handling of Mg2+ binding or dangling ends significantly diverges from what is optimal for the studied RNAs.

      The authors' observation in section 3.1 that "the low percent error exhibited by pKiss could be the result of the pseudoknot 'enforce' constraint, but it is more likely that this outcome was multivariable, equating to the Turner energy model used, and the sensitive auxiliary parameters enforced by the program" is particularly insightful. This highlights the complexity of RNA structure prediction algorithms.

      To build on these findings, a structured comparative analysis of parameter handling across different software tools could be highly beneficial. This analysis would not only clarify why certain engines performed better than others but also help in identifying best practices or potential biases in prediction methodologies. Such an addition would significantly strengthen the study's conclusions and provide valuable guidance for future research in RNA structure prediction.

    • In Section 3.1 of the manuscript, no significant difference in Percent Error was identified. However, it does not specify the statistical test employed nor the method used for adjusting p-values, which are essential details for validating the results. Additionally, the term "Vij" is introduced early in the manuscript but is not contextualized until page 13. Providing this context earlier would enhance the reader's understanding.

    • It would be beneficial if "false positive" and "false negative" were more clearly defined, particularly in the context of mRNA detection. To improve clarity, the authors might consider specifying that sensitivity is the appropriate measure for detecting mRNA among known positives, while specificity is the appropriate measure for detecting mRNA among known negatives, where the probability of false positives is 1-specificity. Additionally, using the Youden Index (J), which is defined as sensitivity + specificity - 1, could provide a helpful summary of detection accuracy. This index ranges from -1 (indicating 100% incorrect detection) to 1 (indicating 100% correct detection), offering a clear metric for assessing performance. (ref: https://www.sciencedirect.com/topics/medicine-and-dentistry/youden-index)

    • Providing the link to the dataset will allow better compliance with open science practices. Please add the link to the dataset as it appears to be missing from the reviewed version of the manuscript. When sharing the dataset, it would be important to also include the associated metadata and appropriate documentation that matches the methods described in the manuscript. For guidelines on how to share data so that it's as reusable as it can be, authors may refer to the FAIR Principles of data sharing (https://www.go-fair.org/fair-principles). 

    • Figure 5B displays PPV as three distinct blocks rather than continuous values, with varying sensitivity within these blocks. This non-random binning of PPV suggests the need for further investigation to understand the underlying causes. 

    • In the discussion section, the authors stated, "We have provided evidence suggesting that MEA software is not always the optimal method of topological prediction when applied to short viral pseudoknotted RNA." This is a significant claim and would benefit greatly from specific references to support the evidence provided in the study. Citing the relevant figures and results that support this claim would significantly enhance comprehension and readability. For example, "As demonstrated in Figure 4, the MEA software Vsfold 5 exhibited higher percent errors in predicting knotted base pairs compared to MFE software like pKiss." Additionally, referencing previous studies that have reported similar findings or that discuss the limitations of MEA methods in RNA structure prediction in the discussion section would strengthen the credibility of the authors' claims by showing that similar limitations have been observed by other researchers. This helps readers understand that the study is building upon existing knowledge. For instance, "Previous studies have also highlighted the limitations of MEA methods in RNA folding predictions, particularly for pseudoknotted structures (in-text citations)."

    List of minor concerns and feedback

    Overall, the reviewers really appreciated how clearly the figures and results were presented. Below are some minor suggested improvements:

    • In the abstract section: please identify the abbreviation (PPV) as Positive Predictive Value

    • Page 3, 1st paragraph after Figure 1: Definitions of pseudoknot should be referenced

    • Page 3, 2nd paragraph after Figure 1: please identify the (NMR) abbreviation as nuclear magnetic resonance

    • Page 7 - The manuscript acknowledges the skewness in the data and provides a rationale for its presence. It's noted that this skewness impacts the training and testing phases, often contributing to false positives and false negatives. It would be beneficial if the authors could elaborate on how they addressed data imbalance, particularly in relation to reducing false positives and false negatives. This additional detail would enhance the understanding of the methods used to manage data skewness and improve model performance.

    • Page 8, 2nd paragraph Mathews et al. 2019 should be corrected to Mathews, 2019 (45)

    • Page 8, Equation (1): add a "%" next to *100 giving the output of x%

    • Page 10, Figure 4:in the title "accurcy" should be corrected to "accuracy"

    • Page 10, Figure 4: bar of the standard deviation (SD) of Vienna (knotted) is not presented

    • Page 10, Figure 4: the bars of the SD seem to be widely large indicating significant variability in the results, so a test of normality of data distribution should be performed before comparisons. This is also observed for kinefold results in Figures 5 & 6

    • Page 12, Figure 6B: the color bar on heatmaps is missing

    Concluding remarks

    One of the authors of the manuscript (VM) was present during the call and provided some additional information regarding the source code which we would like to report here as additional resource for the reader:

    Author's note: "Although original source code was not implemented within this investigation, several well-established web servers were used to generate the data present within this investigation. The link to each web server is will be provided below:

    We thank the authors of the preprint for posting their work openly for feedback. We also thank all participants of the Live Review call for their time and for engaging in the lively discussion that generated this review.

    Competing interests

    Daniela Saderi was a facilitator of this call and one of the organizers. No other competing interests were declared by the reviewers.

  2. Version published to 10.1101/2024.03.21.586060v1 on bioRxiv