RNA-Seq is not required to determine stable reference genes for qPCR normalization

  1. Nirmal Kumar Sampathkumar
  2. Venkat Krishnan Sundaram
  3. Prakroothi S. Danthi
  4. Rasha Barakat
  5. Shiden Solomon
  6. Mrityunjoy Mondal
  7. Ivo Carre
  8. Tatiana El Jalkh
  9. Aïda Padilla-Ferrer
  10. Julien Grenier
  11. Charbel Massaad
  12. Jacqueline C. Mitchell

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

Assessment of differential gene expression by qPCR is heavily influenced by the choice of reference genes. Although numerous statistical approaches have been proposed to determine the best reference genes, they can give rise to conflicting results depending on experimental conditions. Hence, recent studies propose the use of RNA-Seq to identify stable genes followed by the application of different statistical approaches to determine the best set of reference genes for qPCR data normalization. In this study, however, we demonstrate that the statistical approach to determine the best reference genes from commonly used conventional candidates is more important than the preselection of ‘stable’ candidates from RNA-Seq data. Using a qPCR data normalization workflow that we have previously established; we show that qPCR data normalization using conventional reference genes render the same results as stable reference genes selected from RNA-Seq data. We validated these observations in two distinct cross-sectional experimental conditions involving human iPSC derived microglial cells and mouse sciatic nerves. These results taken together show that given a robust statistical approach for reference gene selection, stable genes selected from RNA-Seq data do not offer any significant advantage over commonly used reference genes for normalizing qPCR assays.

Article activity feed

  1. Version published to 10.1371/journal.pcbi.1009868
  2. Review Commons

    Note: This rebuttal was posted by the corresponding author to Review Commons. Content has not been altered except for formatting.

    Learn more at Review Commons

    Reply to the reviewers

    Reviewer #1: **Major comments:**

    The authors state that all the RNA and contaminating DNA was validated and verified with nanodrop and BioAnalyzer which is the correct and accepted approach. However, the following concerns arise with testing reaction efficiency and data analysis:

    Comment 1.

    For reaction efficiency, the standard curves for each reference gene and gene of interest target should be included in the supplemental data. A four point standard curve is the bare minimum to assess reaction efficiency and raises concerns about the data quality. The unknown samples being tested should also be plotted on the corresponding standard curves to assess their efficiency

    Response:

    We have indeed calculated primer efficiencies by serial dilution and performed a four-point standard curve wherever possible. In other cases, at least a three point dilution curve was performed to assess primer efficiency. To have a more extensive range of Cq values in the standard curve, the dilution series was done with serial dilution by a factor of 1/10th as indicated in the materials and methods section under the heading “Amplification Efficiencies”. This provided a range of 6.6 cycles (three point dilution) and about 9.9 cycles (4 point dilution for the primers tested. If the Cq values of the 4th dilution fell beyond the detection range of the machines (above 29 cycles) or closer to the No-RT Control, only the first three dilutions were taken into consideration. We have now included the standard curve for all the genes in the metadata/source data and updated the Figshare DOI. All sample Cq values were within this standard curve as mentioned in the materials and methods section and they have been disclosed already in the metadata files. The raw qPCR Cq output for all references and targets for both datasets can be retrieved from the data file in figshare. Moreover, we will also add a new sentence in the methods sections clarifying the standard curve dilutions and data availability.

    Comment 2. The statement starting on line 510: "The WT experimental group was omitted from this analysis as it was used as the experimental calibrator for differential expression. The mean Fold Change of the WT group is always at 1 regardless of the gene/method in question and therefore it is redundant to test for statistical significance of the WT fold change levels across different methods for each gene." indicates that data analysis was not performed in a rigorous and generally accepted manner. PLease check the analysis with that described in: https://www.cell.com/trends/biotechnology/fulltext/S0167-7799(18)30342-1

    The generally accepted methodology for relative, normalized qPCR data analysis is well described in Figure 5 of that article. qPCR statistical analysis should be performed on the log transformed expression results also well described in that paper.

    Response:

    We apologize for any lack of clarity on this line. We have always compared the Control group to the Test group while performing statistical analysis as shown Figure 3 and Figure 6. This is a fundamental point of any study and we have strictly adhered to this. The highlighted statement pertains to the supplementary figures 1 & 2 where we compare the Fold changes of the “Test” groups between qPCR and RNA-Seq in both datasets. Comparing the Control groups with one another between these methods is redundant as the mean Fold change of the control groups are always 1 as we are measuring relative expression. Thus, we cannot perform any meaningful statistical testing between the control groups between RNA-Seq and qPCR regardless of the method employed for testing.

    Furthermore, the use of the 2-ΔΔCt method for relative expression is in strict adherence to the initial papers describing this method (Livak and Schmittgen 2001, Schmittgen and Livak, 2008), which is again recapitulated in the article that you have cited. This can be seen in the metadata where the excel files that were used for calculating differential expression for all samples and datasets can be accessed. However, we would like to remark that we use more stringent criteria for primer validation (Efficiency between 95% and 105% as opposed to between 90% and 110% as mentioned in the paper). Moreover, the statistical testing and data representation prescribed in Figure 5 of the article that you have mentioned are not well founded for the following reasons:

    • We cannot perform parametric T-Tests using low sample sizes. Furthermore, we cannot test for data normality using few data points employed in standard qPCR assays. Thus, neither our qPCR assays nor the ones used in the mentioned article have enough samples to perform a T-Test. Hence, we have used a non – parametric ordinal Mann Whitney test for testing statistical significance in our study, as it is more apt for such low sample sizes and distributions.
    • The article proposes data representation with the mean and SEM or 95% Confidence Intervals (CI). We would like to kindly remark that SEM and CI are sampling parameters that arise when we perform sampling of data points from a larger population. In our study, we have always shown all the data points (biological replicates) for each experimental group. Hence, we can only show the distribution around the mean with the standard deviations (SD) and not with SEM or CI. We have not performed any sampling whatsoever nor has the study mentioned by the reviewer.

    Comment 3. The authors used Normfinder to assess reference gene stability. Since Normfinder uses a particular algorithm for assessing stability, it is recommended to assess stability using a combination of these "stability calculators" including: GeNorm, NormFinder and BestKeeper. This is described in Table 1 of: https://www.cell.com/trends/biotechnology/fulltext/S0167-7799(18)30342-1. This will give a much more reliable perspective on the ranking of reference genes by their stability.

    Response:

    The method used in our study for reference gene validation is a combination of CV, Normfinder and statistical testing of raw expression profiles. In our previous study (Sundaram et al 2019), we have categorically shown that using a combination of different existing methods such as GeNORM, NormFinder and Best Keeper and comparing their ranks results in a sub optimal choice of reference genes. This is because GeNorm ranks genes with similar expression patterns as stable even if they vary significantly among groups. BestKeeper calculates variation based on Cq values which are exponents while the expression levels are calculated in the linear scale (2^Cq). NormFinder stability scores are influenced by the presence of genes with significant overall variation. More evidence backed up with data can be found in our previous study where we have clearly shown that combining these methods and calculating an overall rank (as proposed in the article you have mentioned) is not the best strategy. Hence, we devised the approach used in the present study, which has been previously validated, published (Sundaram et al 2019, PLoS ONE) and was designed taking into account the advantages and disadvantages of the different existing approaches.

    Comment 4. Finally, since many currently studied targets for relative gene expression are low expressed, it would be important to also examine three deferentially expressed targets in the Cq range of 29 to 32. Yes the variability will be higher but these data will give a more realistic test of reference gene stability.

    Response:

    The target genes used in the study range from about 12 cycles to about 29 cycles (both datasets included, please refer to the source data/metadata). This falls well within the standard curves of all these genes used as mentioned earlier. The stability of the reference genes has been shown with absolute parameters such as the Co-efficient of variation and the Normfinder S scores (Tables 1, 2, 3 & 4). Although we are not opposed to adding more target genes, we fail to see as to how adding target genes with Cq values above 29 cycles would reflect on the stability of reference genes. The variability that will be observed is a mere reflection of the variability of Cq values of the target genes in the Cq range of 29 – 32 as it approaches the detection limits of qPCR assays. The Cq values of the best reference genes would still remain the same. Therefore, this exercise cannot test the “stability” of the reference genes but only demonstrate the limit of qPCR detection (which is already well known). We would also like to remark that we have used No-RT controls in our qPCR assays, which exhibit a signal (different dissociation peak) in this Cq range for some genes and hence this is not a signal that arises from the cDNA. Therefore, we do not consider values above 29 cycles are reliable in our qPCR setup and we switch to droplet digital PCR for such low-expressed genes in our studies.

    Reviewer #2: **Summary + Minor Comments**

    Reference gene selection is one of the most critical steps in gene expression analysis using qPCR. The authors compared data quality using references selected based on RNA-Seq or using panel of often used reference genes. The manuscript is well prepared and easy to understand. Figures are nice and clear. I do not have major comments, but rather a few suggestions to make the manuscript more advanced. Since it is based on already available data or a few more expression measurements could be easily added, I would suggest to include total RNA factor, some rRNA and mtRNA as potential references. It will be interesting to compare their stability and effect on results of other targeted genes.

    In discussion, authors suggested that: "stable reference genes for qPCR data normalisation can be obtained from any random set of candidates provided the statistical approach of reference gene validation is sound and consistent". I do not think the word random in many sentences is appropriate. Panel of reference genes used in this study contains many known stable genes and that does not look random to me. I would rephrase these sentences. Usually panels of reference genes (for human and mouse are commercially available and contains several genes used in study) are composed of genes coding various biological processes to ensure that some of them will be stably expressed in experiments.

    Response:

    We understand the reviewer’s perspective on the use of the words “random reference genes”. We have replaced it with the words “conventional reference genes” throughout the manuscript.

    Regarding the addition of other RNA species as reference genes, we would like to clarify that we have used only protein coding transcripts (encoded by nuclear genes) as reference genes as all our target genes also belong to the same RNA category. This was done in accordance with the MIQE guidelines for qPCR data publication (Bustin et al 2009, DOI: 10.1373/clinchem.2008.112797) which states that rRNA should not be used for mRNA target gene normalization. This is because the vast majority of RNA from total RNA extraction is rRNA and only about 1% - 5% is mRNA. Thus, it is advisable to normalize mRNA targets with mRNA reference genes as it serves as a control for the extraction and RT PCR protocol. This argument can also be extended to other RNA species either in type or in origin (mtRNA). Regarding the total RNA factor, we have always used the same quantity of total RNA from all samples for RT-PCR as mentioned in the materials and methods section.

    Reviewer #3

    **Summary + Minor Comments**

    The aim of this study was to demonstrate that the statistical approach to determine the best reference genes from randomly selected "standard" reference genes might be more sufficient than employing reference genes as indicated by RNA-Seq.

    In a previous study they established a qPCR data normalization workflow, after comparing several statistical approaches for the assessment of reference gene stability. In this study they apply this workflow to compare "random" reference genes with preselected references genes based on RNA-Seq data. They test their hypothesis in two different experimental setups, varying sample material and methodology. After establishing the most "stable" reference genes, the suitability of these genes for normalization was put on trial by investigating their ability to normalize differential expression of target genes. These results were compared to one another and to fold-changes computed from RNA-Seq. The results indicate that as stated in the title of the study, "RNA-Seq is not required to determine stable reference genes for qPCR normalization", since both approaches render similar results. Potential pitfalls when selecting genes from RNA-seq data are discussed and an integration of influencing factors is suggested.

    The key conclusions of the study are convincing and well-supported by the experiments conducted, which are realistic in terms of time and resources. Data and methods are presented articulate and are reproducible. Experiments are adequately replicated and statistical analysis is adequate. The manuscript is well written, tables and figures provided are sound and corroborate a better understanding of the presented results. Minor changes would be:

    Figure 1, 2, 3, 4, 5, 6: in the figure are uppercase letters, in the figure legend are lowercase letters, please adjust that.

    p10 line 347: I understand what is meant, with "using the NF as the reference gene", however, stating again that the combined NF of the two most stable ref genes was used here, would make it clearer. P11 line 355f: the first sentences here are negligible, as already stated elsewhere P30 line 777: The last sentence is not clear to me.

    Response:

    *All minor concerns have been addressed in the revised manuscript as follows: *

    1. Figure 1, 2, 3, 4, 5, 6: in the figure are uppercase letters, in the figure legend are lowercase letters, please adjust that – Has been modified
    2. p10 line 347: I understand what is meant, with "using the NF as the reference gene", however, stating again that the combined NF of the two most stable ref genes was used here, would make it clearer. – Has been modified
    3. P11 line 355f: the first sentences here are negligible, as already stated elsewhere – Have been removed
    4. P30 line 777: The last sentence is not clear to me.

    We wanted to say that our study aptly addressed the strongest hurdle in performing reliable qPCR assays, which is the choice of good reference genes. This choice is not dependent on RNA-SEQ results. We have modified this sentence for better clarity*. *

  3. Review Commons

    Note: This preprint has been reviewed by subject experts for Review Commons. Content has not been altered except for formatting.

    Learn more at Review Commons

    Referee #3

    Evidence, reproducibility and clarity

    Summary

    The aim of this study was to demonstrate that the statistical approach to determine the best reference genes from randomly selected "standard" reference genes might be more sufficient than employing reference genes as indicated by RNA-Seq.

    In a previous study they established a qPCR data normalization workflow, after comparing several statistical approaches for the assessment of reference gene stability. In this study they apply this workflow to compare "random" reference genes with preselected references genes based on RNA-Seq data. They test their hypothesis in two different experimental setups, varying sample material and methodology. After establishing the most "stable" reference genes, the suitability of these genes for normalization was put on trial by investigating their ability to normalize differential expression of target genes. These results were compared to one another and to fold-changes computed from RNA-Seq. The results indicate that as stated in the title of the study, "RNA-Seq is not required to determine stable reference genes for qPCR normalization", since both approaches render similar results. Potential pitfalls when selecting genes from RNA-seq data are discussed and an integration of influencing factors is suggested.

    The key conclusions of the study are convincing and well-supported by the experiments conducted, which are realistic in terms of time and resources. Data and methods are presented articulate and are reproducible. Experiments are adequately replicated and statistical analysis is adequate. The manuscript is well written, tables and figures provided are sound and corroborate a better understanding of the presented results. Minor changes would be:

    Figure 1, 2, 3, 4, 5, 6: in the figure are uppercase letters, in the figure legend are lowercase letters, please adjust that.

    p10 line 347: I understand what is meant, with "using the NF as the reference gene", however, stating again that the combined NF of the two most stable ref genes was used here, would make it clearer. P11 line 355f: the first sentences here are negligible, as already stated elsewhere P30 line 777: The last sentence is not clear to me.

    Significance

    In the last years the necessity of stable reference genes for the normalization of pPCR data has become more and more apparent, since it has been shown, that selecting the genes most "popular", might not always lead to correct expression profiles, since depending on the experimental setup, significant variation can occur. Numerous studies exist, validating potential reference genes, employing several well-established statistical approaches (Genorm, Normfinder etc.) and more recently based on RNA-Seq data. RNA-Seq is definitely accompanied by more work effort and higher costs. Therefore employing the "simpler" approach, obtaining the same results might be beneficial for scientists, establishing a new qPCR protocol, in particular in times, when working cost-effectively is a prerequisite in most laboratories.

    The authors performed a thorough analysis of the two approaches compared in this study. By investigating two entirely different experimental set-ups with a similar outcome, they nicely substantiate their findings. Furthermore, by investigating differential expression of target genes, for both experimental setups, they put their results to the test, convincingly corroborating their results.

    This manuscript is well-written, experiments are thoroughly performed, the findings are convincing and it clearly is an important contribution for the scientific community.

  4. Review Commons

    Note: This preprint has been reviewed by subject experts for Review Commons. Content has not been altered except for formatting.

    Learn more at Review Commons

    Referee #2

    Evidence, reproducibility and clarity

    Summary:

    Reference gene selection is one of the most critical steps in gene expression analysis using qPCR. The authors compared data quality using references selected based on RNA-Seq or using panel of often used reference genes. The manuscript is well prepared and easy to understand. Figures are nice and clear. I do not have major comments, but rather a few suggestions to make the manuscript more advanced. Since it is based on already available data or a few more expression measurements could be easily added, I would suggest to include total RNA factor, some rRNA and mtRNA as potential references. It will be interesting to compare their stability and effect on results of other targeted genes.

    In discussion, authors suggested that: "stable reference genes for qPCR data normalisation can be obtained from any random set of candidates provided the statistical approach of reference gene validation is sound and consistent". I do not think the word random in many sentences is appropriate. Panel of reference genes used in this study contains many known stable genes and that does not look random to me. I would rephrase these sentences. Usually panels of reference genes (for human and mouse are commercially available and contains several genes used in study) are composed of genes coding various biological processes to ensure that some of them will be stably expressed in experiments.

    Significance

    Good reference gene selection is needed for most of experiments, where quantities and qualities of samples are not identical. Unfortunately, every experiment has other stable and reliable reference genes. Validation can be time consuming and expensive. RNA-Seq experiments covering broad spectrum of biological samples are potentially a way for faster identification of unknown stable genes, which could be used for normalization in qPCR. Authors compared effectivity of reference genes selected based on RNA-Seq and using panel of potential reference genes. I like their comparison, but do not fully agree with "random" selection.

    I am not aware of other study comparing quality of qPCR references from RNA-Seq or preselected genes. I think the manuscript will be appreciated by technically or methodically oriented readers (gene expression area).

  5. Review Commons

    Note: This preprint has been reviewed by subject experts for Review Commons. Content has not been altered except for formatting.

    Learn more at Review Commons

    Referee #1

    Evidence, reproducibility and clarity

    This article contrasts RNAseq and random selection to assess reference genes for relative gene expression. The study was well contrived with a solid experimental design.

    Major comments:

    The authors state that all the RNA and contaminating DNA was validated and verified with nanodrop and BioAnalyzer which is the correct and accepted approach. However, the following concerns arise with testing reaction efficiency and data analysis:

    1. For reaction efficiency, the standard curves for each reference gene and gene of interest target should be included in the supplemental data. A four point standard curve is the bare minimum to assess reaction efficiency and raises concerns about the data quality. The unknown samples being tested should also be plotted on the corresponding standard curves to assess their efficiency.
    2. The statement starting on line 510: "The WT experimental group was omitted from this analysis as it was used as the experimental calibrator for differential expression. The mean Fold Change of the WT group is always at 1 regardless of the gene/method in question and therefore it is redundant to test for statistical significance of the WT fold change levels across different methods for each gene." indicates that data analysis was not performed in a rigorous and generally accepted manner. PLease check the analysis with that described in: https://www.cell.com/trends/biotechnology/fulltext/S0167-7799(18)30342-1

    The generally accepted methodology for relative, normalized qPCR data analysis is well described in Figure 5 of that article. qPCR statistical analysis should be performed on the log transformed expression results also well described in that paper.

    The authors used Normfinder to assess reference gene stability. Since Normfinder uses a particular algorithm for assessing stability, it is recommended to assess stability using a combination of these "stability calculators" including: GeNorm, NormFinder and BestKeeper. This is described in Table 1 of: https://www.cell.com/trends/biotechnology/fulltext/S0167-7799(18)30342-1. This will give a much more reliable perspective on the ranking of reference genes by their stability.

    Finally, since many currently studied targets for relative gene expression are low expressed, it would be important to also examine three deferentially expressed targets in the Cq range of 29 to 32. Yes the variability will be higher but these data will give a more realistic test of reference gene stability.

    Significance

    This article will be useful for all labs conducting gene expression experiments. It also uncovers additional contrasts between qPCR and RNA seq which are helpful in choosing the appropriate technology for given experiments.

    Referee Cross-commenting

    I agree with the other reviewers comments.

  6. Version published to 10.1101/2021.08.21.457202v2 on bioRxiv
  7. Version published to 10.1101/2021.08.21.457202v1 on bioRxiv