CovidExpress: an interactive portal for intuitive investigation on SARS-CoV-2 related transcriptomes

  1. Mohamed Nadhir Djekidel
  2. Wojciech Rosikiewicz
  3. Jamy C. Peng
  4. Thirumala-Devi Kanneganti
  5. Yawei Hui
  6. Hongjian Jin
  7. Dale Hedges
  8. Patrick Schreiner
  9. Yiping Fan
  10. Gang Wu
  11. Beisi Xu

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

Infection with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) in humans could cause coronavirus disease 2019 (COVID-19). Since its first discovery in Dec 2019, SARS-CoV-2 has become a global pandemic and caused 3.3 million direct/indirect deaths (2021 May). Amongst the scientific community’s response to COVID-19, data sharing has emerged as an essential aspect of the combat against SARS-CoV-2. Despite the ever-growing studies about SARS-CoV-2 and COVID-19, to date, only a few databases were curated to enable access to gene expression data. Furthermore, these databases curated only a small set of data and do not provide easy access for investigators without computational skills to perform analyses. To fill this gap and advance open-access to the growing gene expression data on this deadly virus, we collected about 1,500 human bulk RNA-seq datasets from publicly available resources, developed a database and visualization tool, named CovidExpress ( https://stjudecab.github.io/covidexpress ). This open access database will allow research investigators to examine the gene expression in various tissues, cell lines, and their response to SARS-CoV-2 under different experimental conditions, accelerating the understanding of the etiology of this disease to inform the drug and vaccine development. Our integrative analysis of this big dataset highlights a set of commonly regulated genes in SARS-CoV-2 infected lung and Rhinovirus infected nasal tissues, including OASL that were under-studied in COVID-19 related reports. Our results also suggested a potential FURIN positive feedback loop that might explain the evolutional advantage of SARS-CoV-2.

Article activity feed

  1. Review Commons

    Note: This rebuttal was posted by the corresponding author to Review Commons. Content has not been altered except for formatting.

    Learn more at Review Commons

    Reply to the reviewers

    Reviewer #1 (Evidence, reproducibility and clarity (Required)):

    **Summary:**

    The manuscript submitted by Djekidel et al entitled: "CovidExpress: an interactive portal for intuitive investigation on SARS-CoV-2 related transcriptomes" reports on a new web portal to search and analyze RNAseq data related to SARS-CoV-2 infections. The authors downloaded and reprocessed data of more than 40 different studies, which is available on the web portal along with all available meta data. The web portal allows to perform numerous differential expression and gene set enrichment analyses on the data and provides publication ready figures. Because of batch effects that could not be removed, the authors do not recommend to analyze data across studies at this point. The authors conclude that the web portal is unique and will allow scientists to rapidly analyze gene expression signatures related to SARS-CoV-2 infections with the potential to make new discoveries.

    **Major comments:**

    Based on the scientific literature, the web portal seems to be an unprecedented resource to search and analyze SARS-CoV-2-related RNAseq data and as such would certainly be a useful resource for the SARS-CoV-2 scientific community. The authors argue that new discoveries are possible by using their web portal in providing use cases. However, the section detailing the analyses the authors did to generate new hypotheses about genes potentially relevant in SARS-CoV-2 infections are very difficult to follow and without more guidance very difficult to reproduce with the web portal. It would require substantial expert knowledge in RNAseq data analysis without more information being provided. It also seems that key candidate genes identified by their analyses have all been studied or identified to be related to SARS-CoV-2 infections, so it is somewhat unclear whether new hypotheses can be generated by the reanalysis of RNAseq datasets, especially because combining the data from different studies is currently not recommended by the authors. The manuscript would benefit from providing fewer use cases but for each of them providing more information on how the portal and which studies were used to generate them and which findings were not described in the publication of the used studies. Some observations in the manuscript are not substantiated with significance calculations (see below). At times, the English writing (grammar) should be improved.

    We thank the reviewer for the positive comments. We suppose the reviewer conclude it need substantial expert knowledge in RNAseq data analysis were due to lacking Video Tutorial. We have now put up several Video Tutorials and more tutorials would be added along later along with users’ feedbacks. We believed this would help ease reviewers’ concern.

    In response to whether new hypothesis can be generated. Sorry if it’s not clear, for all the case studies and our “CovidExpress Reveals Insights and Potential Discoveries”, our portal has provided information not reported by their original publications, as listed below:

    1. Case study #1: The original publication employed a multiomics approach to find the predictor genes between ICU and non-ICU patient. But it’s not obviously to know which genes were mainly due to expression level, which might be due to other data they included (e.g. mass spectrometry data). Our portal allow user to quickly check their expression level and find SESN2 does not have strong expression differences.
    2. Case study #2: We replace this case study with bacterial-susceptibility genes to show such questions could be quickly asked and answered using our portal. Such investigation has not been reported before.
    3. FURIN’s function have been well related to SARS-CoV-2. However, for all reports we could find, they focused on Furin cleavage sites of SARS-CoV-2 or whether FURIN were expressed in the SARS-CoV-2 sensitive tissues. SARS-CoV-2 infection could up-regulate FURIN expression have never been reported before. The study published the data didn’t mentioned FURIN at all. We have made this discovery simply by using CovidExpress portal to find the differential expressed genes and overlap with the literature-based gene list (Supplementary Table S2), we believe more discoveries could be made by users by selecting different data.
    4. If we search OASL AND " SARS-CoV-2" on pubmed, only 5 results shown up indicated it’s under-studied. And none of them indicated OASL could be up-regulated both by SARS-CoV-2 infected lung and Rhinovirus-infected nasal in human. It is not clear to us if we might misunderstand reviewers’ suggestion as “fewer use cases”. Thus, we haven’t removed any use cases, instead we provided more details to help users understand what and how did we made those discoveries not reported by their original studies using CovidExpress.

    At last, we have gone through substantial scientific editing to improve the grammar.

    **Minor comments:**

    Page 6 last sentence: The statement of this sentence is very much what one would expect. It remains unclear whether the authors mean this as a result to validate the processing of the RNAseq data or as a new discovery. Please, clarify.

    We apologize for the confusion. We intended this statement to be a result confirming what we had expected. We have now amended the text to make this point clearer.

    Figure 3A: The violin plots are so tiny that it is impossible to see any trends. It is also difficult to understand which categories one should compare with each other. If there is anything significant to observe, please, add a statistical test and better guide the reader.

    We agree with the reviewer; therefore, we have removed this figure from the paper. The goal of this figure was to demonstrate how to use violin plots for exploratory analysis; however, in this case, the violin plot did not show a clear trend. By using more filtering and other plots (e.g., Figure 3B-C), we believe we now provide better insight.

    Figure 3C: A legend for the color scale is missing. The signal (I guess expression amounts) for SESN2 seems very weak and the same between ICU and non-ICU samples. What is the significance for assigning this gene to the group of genes being upregulated in ICU samples? Also contrary to what the authors state on page 8, SESN2 does not seem to be highly expressed in ICU samples, however, without knowing what the colors represent (fold changes or absolute expression values?) this is somewhat speculative.

    We thank the reviewer for bringing this to our attention. We have now added a legend for the color scale in the revised figure. In Figures 3A-C, we are showcasing how an exploratory analysis can be performed using CovidExpress. As an example, we investigated the expression of the top 20 genes identified by the random forest classifier of Overmyer et al., 2021, as predictors of ICU and non-ICU cases. In the original Overmyer et al. paper, only the general performance metrics of the models are presented (Fig. 6c-g), but the authors do not show the expression patterns of the top predictors. Hence, we demonstrate how CovidExpress can be used to further investigate some questions not explored in the original paper. SESN2 was listed as a top predictor; however, its expression did not vary between ICU and non-ICU samples, as was also observed by the reviewer. We suspect SESN2 was a top predictor due to other data the Overmyer et al. paper included, such as mass spectrometry data. Our statement about SESN2 was not accurately reflected in the figure; therefore, we have rewritten this section to make it clearer.

    Page 9 first sentence: Please, specify what you mean by "starting list". Furthermore, in this paragraph, how do your results compare to the results from the study that you re-analyze here?

    We thank the reviewer for the question. By “starting list,” we meant the top genes from the Overmyer et al., 2021, article as predictors of ICU and non-ICU cases. We have now rewritten this section to make it clearer. We did not expect our results to differ from their data. Our goal was to ask which of their top predictors (by multi-omics data) show a difference in gene expression. When we downloaded their TPM values from their GEO records, the values were very similar overall (see below).

    Figure 3F: Please add labels to your axes and is there a particular reason why in a correlation plot like this one, the y and x axis are not shown with the same range and why does the y axis not start at 0?

    We thank the reviewer for this helpful comment. Our reasoning for presenting the figure in this way is that different genes can have very different expression levels but still be correlated. For example, if gene A expressed 1, 5, and 10 in samples 1,2, and 3, while gene B expressed 100, 500, and 1000 for samples 1, 2, and 3, then their range would be very different but still perfectly correlated (see panel A below). If we draw the x- and y-axes using the same range, this correlation will not be visually obvious (see panel B below).

    This comparison is different from the correlation plots that compare the expression of one gene in different samples. We apologize for the confusion and to avoid misleading readers, we have enlarged the gene names in the Figure labels to ensure that readers notice their differences. We have also added an option to the correlation plot on our portal so that users can choose the optimal format (see below).

    Page 9 second last sentence: It remains unclear which kind of analysis the authors intend to do here and what the starting question is. Please, try to rewrite with less technical terms (i.e. what do you mean by "precalculated contrasts"). In line with this, it remains unclear what Figure 3I is supposed to show. Please, provide some more information to readers who are not RNAseq analysis experts.

    We thank the reviewer for this suggestion. To avoid any misleading claims, we followed Reviewer #2’s suggestion and replaced the coagulation gene list with a filtered gene list from the “Coronavirus disease - COVID-19” KEGG pathway (hsa05171) to showcase how to identify experiments in which this gene signature is enriched or depleted. We also replaced the related figures and text with new results and rewrote this section to avoid using technical terms.

    Figure 3J is somewhat confusing. Why is the mean expression range indicated from 0 to 1 and why are all genes apparently having a mean expression of 1?

    We thank the reviewer for this question. Because the levels of expression of different genes can vary greatly, in Figure 3J (new Figure 3A and 3I), we normalized the mean expression levels of the genes to their maximum values across groups to improve the visualization. We have now made this clearer in the figure, legend, and text.

    Page 10 line 5-6. Are you referring to coagulation markers here or general expression patterns? In case of the latter, how does this statement fit to the paragraph about analyzing expression patterns of coagulation markers? Please, specify. And in line with this, are the highlighted genes in Figure 3K coagulation markers? If not, what is the relevance of these to make the point that one can use the portal to investigate the role of coagulation markers in SARS-CoV-2 infections?

    As mentioned above, to avoid any misleading claims, we followed Reviewer #2’s suggestion and replaced the coagulation gene list with a filtered gene list from the “Coronavirus disease - COVID-19” KEGG pathway (hsa05171). This revision enables us to show how to identify experiments in which this gene signature is enriched or depleted. We have now replaced these figures and text with new results.

    The appearance of describing batch effects and attempts to remove them from the studies was somewhat surprising on page 10 as I would expect this kind of results rather earlier in the results section before describing use cases of the data. You may consider changing the order of your results for a better flow.

    We apologize for the confusion. However, we want to make it clear that the analysis before page 10 did not involve “batch effect”; all analyses were performed within each study. Thus, it is not necessary to change the order in which the results are presented. Also, based on Reviewer #2’s comments, we did not accurately use the term “batch effect,” because “batch effects are purely due to technical differences.” We have now revised the corresponding text to make this point clearer.

    Page 11, second paragraph. Please, explain briefly what the silhouette score is supposed to reflect and thus how Figure S4G should be interpreted. The difference of both bars in Figure S4G is very marginal and thus, does not seem to support the statement of the authors that the ssGSEA scores-based projection is better unless you perform a significance test or I misunderstood. Please, clarify.

    We thank the reviewer for this suggestion. We have now added an explanation of the silhouette score in the manuscript. Briefly, a silhouette score is a metric of the degree of separability of gene clusters from the nearest cluster. For a given sample, lets be the mean intra-cluster distance, and be the mean distance to the nearest cluster. The silhouette score (sil) will be calculated as follows

    The silhouette score ranges between -1 and 1. A value near 1 means that the clusters are well separated, and a value near -1 means that the clusters are intermingled. Using a Wilcoxon rank test, we showed that using ssGSEA scores significantly improves the separability of global GTEx tissues (in Figure S4G; p=8.75e-26).

    Page 11, third paragraph: Figure 4B, to the best of my understanding, does not support the claim that samples clustered less according to study cohorts using the ssGSEA approach. Please, quantify the effect and test for significance or better explain.

    We apologize for the confusion. We quantified the separability between cohorts (GSE ids) by using the silhouette score. In Figure S4H (panel A below), we show that the TPM-based PCA leads to more separation by studies than does the Covid contrast ssGSEA scores in which the separation between studies is less prominent (p-value=0.0045, paired Wilcoxon test).

    For the analyses described starting on page 12 it remains largely unclear whether they were conducted across studies or within studies and which studies were used. This section until the end of the results would especially benefit from providing more information on how the analyses were performed, either in the results or in the methods section.

    We apologize for the confusion. The goal of the analysis on page 12 and the corresponding Figure 4G was to identify genes whose expression increased in both the SARS-CoV-2 infection lung and rhinovirus-infected nasal tissue. Hence, we did a log2(fold-change) vs log2(fold-change) comparison. The log2(fold-change) values were independently calculated for each study. Because we compared values by using the same ranking metric, the cross-samples comparison was possible, as shown in Figure 4G. We have now added more details to the Methods section to clarify this point.

    Figures 4J and 4K miss axis labels and since we look at correlations, the figures could be redrawn using the same ranges on x and y axis.

    We thank the reviewer for this suggestion. We have now added axes labels to the new figures. However, we have not used the same range on the x and y axes because they depict expression levels of different genes. For example, if gene A is expressed 1, 5, and 10 in samples 1, 2, and 3, while gene B is expressed 100, 500 and 1000 for samples 1, 2, and 3, their range would be very different but still perfectly correlated (panel A below). If we draw x and y axes using the same range, this correlation will not be visually obvious (panel B below).

    This comparison is different from the correlation plots that compare the expression of one gene in different samples. We apologize for the confusion and to avoid misleading readers, we have enlarged the gene names in Figure labels to ensure that readers notice they are different genes. We have also added an option to the correlation plot on our portal so that users can choose the optimal format (see below).

    Page 14 line 5: Is this the right figure reference here to Figure 4G? If yes, then it is unclear how Figure 4G supports the statement in this sentence. Please, clarify.

    We apologize for the confusion. In Figure 4G, we labeled several important genes and used different colors to indicate whether the gene was regulated by SARS-CoV-2 only (purple), Rhinovirus only (black), or both(red). FURIN was the gene that is only significantly upregulated by SARS-CoV-2. The data in Figure 4G were from GSE160435(“SARS-CoV-2 infection of primary human lung epithelium for COVID-19 modeling and drug discovery”); that study used lung organoid alveolar type 2 (AT2) cells as the model. We think this confusion was caused by our failure to provide the details about the GSE160435 study. We have now amended the manuscript to include these details in the Methods section to avoid confusion. We also enlarged the gene labels in the figure to make them more visible. In the manuscript, we have changed from “our results found FURIN gene was also upregulated in SARS-CoV-2–infected lung organoid alveolar type 2 cells (Figure 4G, Supplementary Table S3).” to “We found that FURIN was upregulated in SARS-CoV-2-infected lung organoid alveolar type 2 cells (Figure 4G, Supplementary Table S4) (Mulay, Konda et al., 2021), it has reported that TGF-β signaling could also regulates FURIN (Blanchette, Rivard et al., 2001). Our gene enrichment analysis also found TGF-β signaling enriched only for up-regulated genes in SARS-CoV-2-infected lung cells (FDR correct p=7.58E-05, Supplementary Table S4), these observations implicated a positive feedback mechanism only for SARS-CoV-2-infected lung but not RV-infected nasal cells.”

    Figure 2 is of too low resolution. Many details cannot be read. Please, provide a higher resolution figure.

    We apologize for the inconvenience. However, we did not expect the reader to read the details on Figure 2, as it is just an overview of the CovidExpress portal. The aim is give the reader an impression about what functions CovidExpress could offer.

    Reviewer #1 (Significance (Required)):

    Providing a single platform for the analysis of SARS-CoV-2-related RNAseq data is certainly of high value to the scientific community. However, as the portal and manuscript are currently presented, for scientists that are not RNAseq analysis specialists, more guidance would be required to understand and use correctly the functionalities of the portal. Unfortunately, because batch effects could not be removed from the studies, the authors, correctly, do not recommend to combine data from different studies for analyses, however, this likely will also limit the potential of the resource to make new discoveries beyond what the original studies have already published. As indicated above, the authors could support their claim by comparing their findings with findings published from the studies they reanalyzed. The portal is only of use to scientists studying SARS-CoV-2. I am not an expert in RNAseq data analysis and thus cannot comment on the technicalities, especially the processing of the RNAseq datasets. We thank the reviewer for the positive comments. We apologize for the confusion and acknowledge that we should not describe our effort using the term “batch effect.” As described by Reviewer #2 (and we agree), batch effect should be used only to indicate a purely technical difference in the same biological system; for example, differences in experiments performed on different days or by different lab personnel. Thus, we cannot correct for “batch effect” by using CovidExpress. We hope that the reviewer realizes that what we did was correct for the effect caused by differences in software and parameters across the studies. For example, in our approach, the DEGs from GSE155518 and GSE160435 (both primary lung alveolar AT2 cells (both from Mulay et al., Cell Report, 2021) were significantly correlated (panel A below; p = 1.36e-24, F-test). However, when we downloaded the TPM values from their GEO records, GSE155518 appeared to have a genome-wide decrease in the expression of SARS-CoV-2–infected samples (panel B below). We suspect that this is because in their data processing, the expression of virus themselves were also considered. Thus, using the proceed data directly without careful reviewing the method might lead to false hypothesis.

    At last, researchers can make new discoveries, such as our OASL and FURIN findings, by using many other features that CovidExpress provides.

    Reviewer #2 (Evidence, reproducibility and clarity (Required)):

    Djekidel and colleagues describe a web portal to explore several SARS-CoV-2 related datasets. The authors applied a uniform reprocessing pipeline to the diverse RNA-seq datasets and integrated them into a cellxgene-based interface. The major strengths of the manuscript are the scale of the compiled data, with over one thousand samples included, and the data portal itself, which has useful visualization and analysis functions, including GSEA and DEG analysis. My primary concerns with the study are centered on the analysis examples that are presented and their interpretation, as well as the user interface for the data portal.

    **Major Comments:**

    1. The literature analysis feels out of place and is not informative (Fig 1E), as the conclusions that can be drawn from literature mining are minimal. In evidence of this, the authors highlight that CRP is a top-studied "gene" and later voice their interest in how CRP is not a differentially expressed gene (pg6). This illustrates the problems with the literature-based analysis, since in the context of COVID-19, CRP is a common blood laboratory measurement that is used as a general marker of inflammation. Transcription of CRP is essentially exclusively in hepatocytes as an acute phase reactant (see GTEx portal for helpful reference), and would therefore not be expected to be found in the various datasets collected by the authors. The one exception might be liver RNA-seq samples from COVID-19 patients, but I do not think these are available in the current collection. I would therefore suggest to remove the literature analysis parts from the manuscript.

    We thank the reviewer for sharing knowledge about CRP. As discussed in our manuscript, we agree that not all top genes from literature-based analysis were expected to be included in RNA-seq analysis. We apologize for the confusion, and we have amended our description to make this point clearer. However, we still believe that literature-based analyses are very useful in the following aspects:

    1. This type of analysis bridges the gap between data-driven research and hypothesis-driven research. For example, we found many genes in our meta-analysis, but it is not feasible to describe the functions of all of them. Thus, in Figure 1F, we color-coded genes in red if they also appeared as top genes in the literature-based analysis and read related manuscripts to build confidence that the meta-analysis is useful. Then we expanded our review to more top genes and found more interesting evidence (Supplementary Table S2, “TopGenesbyDifferentialAnalysis” tab).
    2. Literature-based analyses also reduce the time researchers spend prioritizing their investigations. For example, in our comparison of SARS-CoV-2–infected lung and Rhinovirus-infected nasal tissue, we found >2000 genes upregulated only in SARS-CoV-2–infected lung but not in Rhinovirus-infected nasal cells. It is not easy to derive a hypothesis from so many genes. When we overlapped the gene list with literature-based analysis, FURIN popped up as the most well-studied gene, and we did not find any report that mentioned that SARS-CoV-2 can regulate FURIN This raised our interest and led to a suggested mechanism in which SARS-CoV-2 could evolve to induce FURIN expression and gain superior infectivity. FURIN’s upregulation is significant but not among the top genes, in terms of fold change (>2-fold change, FDR p th by fold change). Thus, without the literature-based analysis, this observation could have easily been neglected.
    3. Such analyses help researchers to prime their hypotheses for novel findings. For example, in our comparison between SARS-CoV-2–infected lung and Rhinovirus-infected nasal tissues (Figure 4G, Supplementary Figure 5D and E), we found many upregulated genes, but OASL was not in our literature-based analysis, which indicated that it is under-studied and worth highlighting. We hope the reviewer will agree that we should retain the literature-based analysis in our paper. These analyses were not meant to be conclusive but rather a way to prioritize investigations. Finally, we removed CRP from Fig 1E and the main text to avoid confusion.
    1. The data portal, implemented through cellxgene, is accessible for non-programmers to use. However, it is very easy to end up with an "Unexpected HTTP response 400, BAD REQUEST" error, with essentially no description of the cause of the error or how to rectify it. When this occurs (and in my experience it occurs very frequently), this also forces the user to refresh the page entirely, losing any progress they may have made. I see that the authors describe this error in their FAQ page, but their answer is not very intuitive and I was unsure of what they meant: "This happens because the samples you selected doesn't contain all "Group by" you want compare for each "Split by" group. You could confirm using the "Diff. groups" buttons.".

    We apologize for the confusion. This excellent point made by the reviewer required an improvement in the software engineering, which we have now completed. We have figured out how to avoid this error and have run thorough tests to ensure that it does not appear anymore. We also added a gitter chat channel to our landing page, so that users can report if they encounter this or other errors.

    I would therefore ask that the authors provide more detailed tutorials (ideally step-by-step) on common analyses that users will want to perform, hopefully minimizing the amount of frustration that users will encounter.

    We thank the reviewer for this suggestion. We have uploaded several video tutorials to our landing page and will gradually add more. We also added a gitter chat channel, so users can ask questions, report bugs, or suggest new studies to include in the portal.

    1. Selection of samples is not very quick or intuitive. If I wanted to select only the samples from one specific GEO accession, I had to resort to individually checking the boxes of the sample IDs that I wanted. If I instead selected the GEO accession under the samples source ID, then used the "Subset to currently selected samples" button, I invariable got the HTTP error 400 message. Of course, this may simply reflect my lack of familiarity with cellxgene; I would nevertheless encourage the authors to improve the FAQ to include a step-by-step example for how to do common analyses/procedures.

    We apologize for the confusion. To select an individual GEO accession, users can simply tick the box beside “Samples Source ID.”

    Then all boxes would be clear for “Samples Source ID” that allow you to select only the one you want. We also have uploaded video tutorials to help users learn how to navigate the portal.

    We apologize for the “HTTP error 400” messages. We figured out that users would encounter that message frequently after they encounter it once due to a back-end cache mechanism. We have now improved the portal from the software-engineering side. In our recent tests of the latest version, this error does not appear anymore. We also added a gitter chat channel on our landing page so that users can report encountering this or other errors.

    1. The second case study, centered on coagulation genes, is misguided. Alteration of coagulation lab values in severe COVID-19 patients is reflecting the general inflammatory state of these patients, and would not be expected to manifest on the transcriptional level in infected cells/tissues. Coagulation labs are measuring the functional status of the coagulation cascade, which is far-removed from the direct transcription of the corresponding genes - proteolytic processing of clotting factors, etc. As with CRP (see above comment), most clotting factors are transcribed almost exclusively in the liver (check GTEx portal); I would not expect upregulation of coagulation factors in lung cell lines/organoids/cultures etc after infection with SARS-CoV-2. I would recommend the authors to pick a different gene ontology set for a case study, as the current one focusing on coagulation is confusing in a pathophysiologic sense.

    We thank the reviewer for this suggestion. To avoid any misleading claims, we have replaced the coagulation gene list with a filtered gene list from the “Coronavirus disease - COVID-19” KEGG pathway (hsa05171) to showcase how to identify experiments in which this gene signature is enriched or depleted. We also replaced Figures 3G-J with new results.

    1. The two large clusters of blood-derived samples vs other tissues is not surprising and the authors' interpretation is confusing. The authors write that "the COVID-19 signature was not able to overcome the tissue specificity and that immune cells might respond to SARS-CoV-2 differently." This should be immediately obvious given the pathophysiology of COVID-19 infection; the cell types that are directly infected by SARS-CoV-2 will of course have a distinct response compared to the circulating blood cells of COVID-19 patients, which are responding by mounting an immune response. There is no reason to expect a priori that the DEGs in the directly infected lung cells would be similar to that of immune cells that are mounting a response against the virus.

    We thank the reviewer for these comments. We agree that it should be obvious that directly infected lung cells would differ from immune cells. However, this has never been shown in a large dataset. Also, it is not obviously whether all other different tissues would respond to SARS-CoV-2 differently. Thus, we believe it is important to present this overview. We have amended the description to deliver clearer message as “This confirmed immune cells respond to SARS-CoV-2 differently from other tissues also suggested the response of most other tissues might sharing similar features.”.

    1. The authors devote considerable space in the manuscript to exploring "batch effects" and trying to minimize them (pg10-11 Fig 4A-D, Fig S4). However, given that the compiled datasets are from entirely different experimental and biological systems (e.g. in vitro infection vs patient infection, different cell lines, timepoints after virus exposure, diverse tissues, varying disease severity), it is inappropriate to simply refer to all of these differences as "batch effects" alone. Usually, the term "batch effect" would refer to the same biological experiment/system (i.e. A549 cells infected with CoV vs control), but performed on different days or by different lab personnel - in other words, batch effects are purely due to technical differences. This term clearly does not apply when comparing samples from entirely different cell lines, or tissues, etc, and the authors should not keep describing these differences as batch effects that should be "corrected" out.

    We thank the reviewer for the insight. We apologize for the confusion caused by using the phrase “batch effect correction” to describe our approach. We agree that the difference between studies should not be referred to as a “batch effect correction” and have now amended the descriptions to avoid confusion.

    Indeed, the authors themselves state that the main point of their "batch effect correction" efforts is only for PCA visualization. I therefore feel this section contributes very little to the overall manuscript, especially given the authors' own recommendation that all analyses should be performed on individual datasets (which I certainly agree with). I assume that the authors were required to provide some sort of dimensional reduction projection for the cellxgene browser, but this is more a quirk in their choice of platform for the web portal. Thus, this section of the manuscript should be deemphasized.

    We thank the reviewer for these comments and again apologize for the confusion caused by our use of the term “batch effect correction” to describe our approach. However, we believe these parts of the paper should be retained for the following reasons:

    • In practice, sample mislabeling can happen. PCA or simple clustering approaches are very useful for helping raise researchers’ attention, so they could further check the possibility of sample mislabeling.
    • Even within a study, one sample can be an outlier due to low or unequal sample quality. Removing outliers would help boost the significance of real findings. Without our approach, it would be harder for users to notice and remove outliers from their investigations.
    • Finally, these efforts are useful for generating hypotheses. For example, although we collected a lot of data, it is not feasible for us to read all the details in all the manuscripts published. We observed a similarity between SARS-CoV-2–infected lung samples and Rhinovirus–infected nasal samples by exploring our portal’s capabilities (Figure 3E-F). Then we read the manuscripts in which those data were published and found that our discovery was consistent with the original studies’ results. We believe these efforts are essential to help researchers generate or refine their hypotheses. As we update the database with more samples, this approach will become increasingly powerful.
    1. Given the limitations of any combined multi-dataset analyses, one very useful feature would be to conduct "meta-analyses" across multiple datasets. For instance, it would be informative to find which genes are commonly DEGs in user-selected comparisons, calculated separately for each dataset and then cross-referenced across the relevant/user-selected datasets.

    We thank the reviewer for this comment. Indeed, we agree that “meta-analyses” are useful and have now compiled Supplementary Table S2 and Figure 1F to demonstrate the commonly regulated genes. To enable user-selected comparisons across studies on our portal, we need to design a thoughtful user interface. Otherwise, the results from our portal could easily cause fatal misinterpretation. For example, GSE154613 includes samples like DMSO, Drug, SARS-CoV-2, and DMSO+SARS-CoV-2. If a user simply selected to compare SARS-CoV-2 versus Control, the results would be SARS-CoV-2 and DMSO+SARS-CoV-2 versus DMSO and Drug. Such functions need time to design and implement; therefore, we will consider this suggestion for further development of our portal.

    **Minor comments:**

    1. Fig S1G, color legend should be added (I understand that these colors are the same from S1H).

    We thank the reviewer for the comment. We have now added information about the colors in the figure legend.

    1. Mouseover text for trackPlot on the data portal is incorrect (it says the heatmap text instead).

    We thank the reviewer for this comment. We have now corrected this bug.

    1. Abstract should be revised to describe only the 1093 final remaining RNA-seq samples after filtering/QC steps.

    We thank the reviewer for this comment. We have now amended the Abstract to include this information.

    1. Text in many figures is too small to be legible. I would suggest pt 6 font minimum for all figure text, including the various statistics in the figure panels.

    We thank the reviewer for this comment. We have now amended the font sizes and will provide high-resolution figures in revision.

    1. Are the DE analyses in Fig 1F specifically limited to control vs SARS-CoV-2/COVID-19 comparisons? Many of the samples included in this study are from other respiratory infections (labeled "other" in Fig 1B).

    We thank the reviewer for the question. Figure 1F was not originally limited to control vs SARS-CoV-2/COVID-19 comparisons, because we thought control vs virus, drug vs mock, or difference between time points would also be interesting. If we narrow the analysis to contrasts only between control vs SARS-CoV-2/COVID-19, Figure 1F would be still look similar (as below) because the genes in that comparison comprise the largest share of genes included in the original graphic.

    In the end, we replaced Figure 1F to avoid confusion and added more details in the Methods.

    1. The word cloud format is not conducive for understanding or interpretation. It would be much more informative to simply have a barplot or similar to clearly indicate the relative "abnudance" of a given gene among all 315 DE analyses.

    We thank the reviewer for this comment but respectfully disagree with this point. Visualization of the relative “abundance” of genes with word clouds is a relatively novel concept in computational biology. However, we believe, that in this case, it has certain advantages over visualization using traditional bar plots for example. The word cloud format allows us to highlight genes relative to their importance, with the word “importance” being used here in the sense of combined metrics from DEGs, as shown in Figure 1F, or the frequency with which genes are mentioned/discussed in various literature sources, as shown in Figure 1E. For this purpose, the exact values will most likely not be important for most users/readers. Be presenting a word cloud visualization, readers can easily discern the top genes and use them in the exploration of their own data or the CovidExpress portal. However, if users want to analyze raw values, we provide in Supplementary Table S3 a full list of all genes and gene sets that can be download from our landing page (section “CovidExpress Expression Data Download”) in GMT format. Also, when we visualized the ranks of genes by using bar plots as the reviewer suggested, the results were much harder to read (as shown in the bar graph below) than simply looking at the raw data in supplementary tables.

    1. Claims of increased/decreased dataset separability should have statistical analysis on the silhouette score boxplots (Fig S4G-I).

    We thank the reviewer for the reminder. We have added statistical tests to referred silhouette score boxplots (Wilcoxon rank test)

    1. Regarding Fig 4E-F - what are the key genes that contribute to PC1, and how do they relate to the DEGs in Fig 4G?

    We thank the reviewer for this question and apologize for the confusion. In Figure 4E-F, the PCA were based on ssGSEA score, as each gene set would have a score for a sample, not individual genes. Thus, the top contributed to PC1 were gene sets upregulated or down-regulated in certain contrasts. We provided on the portal’s landing page detailed results for top gene sets (for the ssGSEA approach) and genes (for the TPM approach) that contributed to various PCs (“Clustering Results for Reviewing and Download” section). This allows users to download and further explore these data.

    1. Statistics describing the relation between OASL And TNF/PPARGC1A should be included to justify the author's statements. This could be correlation, mutual information, regression, etc.

    We thank the reviewer for this suggestion, and we have updated Figures 4J-K to show the correlation values and corresponding F-statistics. The Pearson correlation between OASL and TNF was significant (Pearson Correlation=0.75 and p-value = 6.85e-72), but the correlation between OASL and PPARGC1A had a negative slope and showed a moderately significant p-value (Pearson Correlation=-0.08 and p-value=0.12), confirming to a certain degree our statement. We have now updated the corresponding text in the manuscript.

    1. There are several studies now that have performed scRNA-seq on the lung resident and peripheral immune cells of COVID-19 patients. To more definitively tie in their analyses in Fig 4J-K/Fig S5D-E (to affirm "its important role in the innate immune response in lungs"), the authors should assess whether OASL is upregulated in the lung macrophages of COVID-19 patients vs controls.

    We thank the reviewer for this suggestion. Indeed, Liao, et al. recently reported “BALFs of patients with severe/critical COVID-19 infection contained higher proportions of macrophages and neutrophils and lower proportions of mDCs, pDCs, and T cells than those with moderate infection.” (Nature Medicine, 2020, https://doi.org/10.1038/s41591-020-0901-9). They further refined macrophage data into subclusters and reported top enriched GO terms as “response to virus” (group 1), “type I interferon signaling pathway” (group 2), “neutrophile degranulation” (group 3), and “cytoplasmic translational initiation” (group 4). When we investigated their data, we found that group1 and group2 both identified OASL as a marker gene, indicated OASL might response to virus and help type I interferon signaling. Furthermore, another data set (from Ren et al., Cell, 2021, https://dx.doi.org/10.1016%2Fj.cell.2021.01.053) showed several clusters in patients with severe COVID-19 (left panel below) that were enriched for OASL expression(right panel below).

    We have now added these observations to strengthen our hypothesis about the role of OASL.

    1. The visualization and analysis functions in the data portal appear to work reasonably well out of the box. However, the download buttons for plots did not work in my hands. I realized that a workaround is to right click -> "Save image as" (which then downloads a .svg file), but this is not ideal and should be fixed to improve usability. I had tested the data portal on both Firefox and Edge browsers, using a Windows 10 PC.

    We agree with the reviewer. Due to some technical issues with the figure javascript plugin, the download feature does not work unless the figure is saved as a file on the server side. To avoid any security issues, we tried to minimize new file generations, hence, for the moment we have disabled this feature. Users can still download high-resolution .svg figures by using the right-click -> “save image as.” This information is now included in the FAQ section on the portal’s landing page.

    Reviewer #2 (Significance (Required)):

    The data portal appears to have useful analysis and visualization features, and the data collection appears to be quite comprehensive. I would strongly encourage the authors to continue collecting datasets as they become available and further improving the usability of the portal. As noted in the above comments, I think there is potential for their cellxgene-based browser to be useful to non-computational biologists, but at present, the data portal is not as simple to use as it should be. With further efforts to developing step-by-step tutorials for common analysis/visualization tasks, more informative case studies, and the other revisions suggested above, this study could be a valuable resource for the community. Of note, this review is written from the perspective of a primary wet-lab biologist with extensive bioinformatics experience but limited web development expertise.

    We thank the reviewer for the positive comments. We understand the importance of data updating. Our plan is to complete quarterly updates once this manuscript has been accepted or when 10 new studies have been either collected by us or suggested by users. This information is also now included in the FAQs of the portal’s landing page. We have also uploaded several tutorials videos to the landing page and will gradually add more. We also added a gitter chat channel, so users can ask questions, report bugs, or suggest new studies to add to the database.

    **Referee Cross-commenting**

    I agree with the comments of the other reviewers.

    Reviewer #3 (Evidence, reproducibility and clarity (Required)):

    **Summary:**

    The ongoing COVID-19 pandemic is a big threat to human health. The researchers have conducted studies to explore the gene expression regulations of human cells responding to COVID-19 infection. A website that integrating those datasets and providing user-friendly tools for gene expression analysis is a valuable resource for the COVID-19 study community. The authors collected published RNASeq datasets and developed a database and an interactive portal for users to investigate the gene expression of SARS-CoV-2 related samples. This website would be of great value for the SARS-CoV-2 research community if the batch normalization problems are solved.

    **Major comments:**

    1. The major concern of CovidExpress is the batch effects from different studies. As the authors have shown and mentioned in their discussion that "For the current release, we strongly suggest investigators to perform gene expression comparison within individual study." This limits the usage of CovidExpress as integrating analysis from multiple datasets of different studies is the key value and purpose of CovidExpress.

    We thank the reviewer for the comment. Reviewer #2 reminded us, and we agree, that differences between studies should not be considered “batch effects.” We apologize for the confusion. The GSEA function provided in the portal does not suffer from batch effect, because all the pre-ranked lists of genes are based on contrasts from the same studies. Although we cannot correct for the differences between studies, we did correct for effect caused by differences in software and parameters used. For example, in our approach, the DEGs from GSE155518 and GSE160435 (both studies of primary lung alveolar AT2 cells from Mulay et al., Cell Report, 2021) were significantly correlated (below panel A, p-value = 1.36e-24, F-test). However, if we simply download the TPM values from their GEO records, GSE155518 appears to show a genome-wide decrease in expression in SARS-CoV-2–infected samples (below panel B). These errors might lead to false hypotheses.

    1. The authors should include experimental protocols as one key parameter in the description and further integrating analysis of different datasets. As the authors showed that QuantSeq is a 3' sequencing protocol of RNA sequencing. However, it is not convincing to me that simply excluding QuantSeq samples is the ideal solution for downstream integrating analysis as QuantSeq has been shown that it has pretty good correlations with normal RNASeq methods in gene quantifications. It is interesting that there are 21.2% of samples were biased toward intronic reads. What protocol differences or experimental variations would explain the biases?

    We thank the reviewer for the comment and apologized for not being clearer. One of our main goals re-processing all samples is to correct for pipeline processing–related batch effects. We tried to reduce those effects introduced by using different software or parameters. QuantSeq or similar protocols are heavily bias to 3’ UTR; thus, the software and parameters used for RNA-seq data will not be suitable. In contrast, we agree that the downstream results from QuantSeq have good correlation to RNA-seq (we observed a correlation of ~0.75, when compared to the log2 fold-change from Quant-Seq to RNA-seq). However, we could not reconcile QuantSeq always correlated well with RNA-seq, in terms of individual quantification. For example, Jarvis et al. recently reported only ~0.35 correlation between QuantSeq and RNA-seq (https://doi.org/10.3389/fgene.2020.562445). Theoretically, the correlation would be weaker for genes with a small 3’ UTR. Thus, we will not include QuantSeq data in this portal. However, if we collect enough studies in the future, we will consider uploading a separate portal just for QuantSeq using a pipeline optimized for protocol bias to 3’ UTR.

    For the 21.2% samples that were biased towards intronic reads, we believe they reflect differences in the kits used. For example, of the 162 samples “BASE_INTRON (%)” >30% (Supplementary Table S1) that passed QC, 76 samples were total RNA obtained using the SMARTer kit and 36 were total RNA obtained using the Trio kit. Given that we have 105 samples of total RNA derived using the SMARTer kit and 38 samples of total RNA derived using the Trio kit, we conclude that the Trio kit was more biased toward introns, and the SMARTer kit was also strongly biased. This finding is consistent with those of others who have reported the bias of the SMARTer kit (Song et al., https://doi.org/10.1186/s12864-018-5066-2). Users can find these results in our Supplementary Table S1. We have also uploaded the protocol information to our portal.

    1. How do the authors plan to update and maintain CovidExpress?

    We thank the reviewer for this question. We understand the importance of data updating. Our plan is to update the database quarterly once this manuscript has been accepted or when 10 new studies have been collected by us or suggested by users. We have added this information to the FAQs on the portal’s landing page. We also understand the importance of maintaining the service for a feasible amount of time for research. Therefore, we will keep the server activated for at least 2 years after the WHO announces that COVID-19 is no longer a global pandemic. We will also ensure that, even after we take down the server , scientists with programming skills will be able to create local servers based on the data provided on CovidExpress.

    **Minor comments:**

    1. Some texts in figures are not readable. For example, Fig2B, 2C, 2D, 2E.

    We thank the reviewer for this comment. We have now increased the font sizes and provided high-resolution figures in revision.

    1. The authors could use Videos to demonstrate how to use CovidExpress on the website as they have shown in Fig3.

    We thank the reviewer for this suggestion. We have uploaded several video tutorials to the landing page and will gradually add more. We also added a gitter chat channel so that users can ask questions, report bugs, or suggest new studies to include in the database.

    Reviewer #3 (Significance (Required)):

    The ongoing COVID-19 pandemic is a big threat to human health. Many molecular and cellular questions related to COVID-19 pathophysiology remain unclear and many researchers have conducted studies to explore the gene expression regulations of human cells responding to COVID-19 infection. However, there is no database/website that integrating all RNASeq data to provide user-friendly tools for gene expression analysis for COVID-19 researchers. The authors collected the published RNASeq datasets and developed a database and an interactive portal, named CovidExpress, to allow users to investigate the gene expressions response to COVID-19 infection. CovidExpress is a valuable resource for the COVID-19 study community once the batch normalization problems are solved. The users who came up with ideas about the regulation of COVID-19 response could use the system to test their hypothesis, without experience in bioinformatics and RNASeq data analysis. This will be more important when more RNASeq data from samples with different tissues, cell lines, and conditions are integrated into the database.

    We thank the reviewer for the positive comments. We apologize for the confusion and acknowledge that we should not describe our effort using the term “batch effect.” As described by Reviewer #2 (and we agree), batch effect should be used only to indicate a purely technical difference in the same biological system; for example, differences in experiments performed on different days or by different lab personnel. Thus, we cannot correct for “batch effect” by using CovidExpress. We hope that the reviewer realizes that what we did was correct for the effect caused by differences in software and parameters across the studies. For example, in our approach, the DEGs from GSE155518 and GSE160435 (both primary lung alveolar AT2 cells (both from Mulay et al., Cell Report, 2021) were significantly correlated (panel A below; p = 1.36e-24, F-test). However, when we downloaded the TPM values from their GEO records, GSE155518 appeared to have a genome-wide decrease in the expression of SARS-CoV-2–infected samples (panel B below).

    Thus, using the proceed data directly without careful reviewing the method might lead to false hypothesis. At last, researchers can make new discoveries, such as our OASL and FURIN findings, by using many other features that CovidExpress provides.

  2. Review Commons

    Note: This preprint has been reviewed by subject experts for Review Commons. Content has not been altered except for formatting.

    Learn more at Review Commons

    Referee #3

    Evidence, reproducibility and clarity

    Summary:

    The ongoing COVID-19 pandemic is a big threat to human health. The researchers have conducted studies to explore the gene expression regulations of human cells responding to COVID-19 infection. A website that integrating those datasets and providing user-friendly tools for gene expression analysis is a valuable resource for the COVID-19 study community. The authors collected published RNASeq datasets and developed a database and an interactive portal for users to investigate the gene expression of SARS-CoV-2 related samples. This website would be of great value for the SARS-CoV-2 research community if the batch normalization problems are solved.

    Major comments:

    1. The major concern of CovidExpress is the batch effects from different studies. As the authors have shown and mentioned in their discussion that "For the current release, we strongly suggest investigators to perform gene expression comparison within individual study." This limits the usage of CovidExpress as integrating analysis from multiple datasets of different studies is the key value and purpose of CovidExpress.

    2. The authors should include experimental protocols as one key parameter in the description and further integrating analysis of different datasets. As the authors showed that QuantSeq is a 3' sequencing protocol of RNA sequencing. However, it is not convincing to me that simply excluding QuantSeq samples is the ideal solution for downstream integrating analysis as QuantSeq has been shown that it has pretty good correlations with normal RNASeq methods in gene quantifications. It is interesting that there are 21.2% of samples were biased toward intronic reads. What protocol differences or experimental variations would explain the biases?

    3. How do the authors plan to update and maintain CovidExpress?

    Minor comments:

    1. Some texts in figures are not readable. For example, Fig2B, 2C, 2D, 2E.

    2. The authors could use Videos to demonstrate how to use CovidExpress on the website as they have shown in Fig3.

    Significance

    The ongoing COVID-19 pandemic is a big threat to human health. Many molecular and cellular questions related to COVID-19 pathophysiology remain unclear and many researchers have conducted studies to explore the gene expression regulations of human cells responding to COVID-19 infection. However, there is no database/website that integrating all RNASeq data to provide user-friendly tools for gene expression analysis for COVID-19 researchers. The authors collected the published RNASeq datasets and developed a database and an interactive portal, named CovidExpress, to allow users to investigate the gene expressions response to COVID-19 infection. CovidExpress is a valuable resource for the COVID-19 study community once the batch normalization problems are solved. The users who came up with ideas about the regulation of COVID-19 response could use the system to test their hypothesis, without experience in bioinformatics and RNASeq data analysis. This will be more important when more RNASeq data from samples with different tissues, cell lines, and conditions are integrated into the database.

  3. Review Commons

    Note: This preprint has been reviewed by subject experts for Review Commons. Content has not been altered except for formatting.

    Learn more at Review Commons

    Referee #2

    Evidence, reproducibility and clarity

    Djekidel and colleagues describe a web portal to explore several SARS-CoV-2 related datasets. The authors applied a uniform reprocessing pipeline to the diverse RNA-seq datasets and integrated them into a cellxgene-based interface. The major strengths of the manuscript are the scale of the compiled data, with over one thousand samples included, and the data portal itself, which has useful visualization and analysis functions, including GSEA and DEG analysis. My primary concerns with the study are centered on the analysis examples that are presented and their interpretation, as well as the user interface for the data portal.

    Major Comments:

    1. The literature analysis feels out of place and is not informative (Fig 1E), as the conclusions that can be drawn from literature mining are minimal. In evidence of this, the authors highlight that CRP is a top-studied "gene" and later voice their interest in how CRP is not a differentially expressed gene (pg6). This illustrates the problems with the literature-based analysis, since in the context of COVID-19, CRP is a common blood laboratory measurement that is used as a general marker of inflammation. Transcription of CRP is essentially exclusively in hepatocytes as an acute phase reactant (see GTEx portal for helpful reference), and would therefore not be expected to be found in the various datasets collected by the authors. The one exception might be liver RNA-seq samples from COVID-19 patients, but I do not think these are available in the current collection. I would therefore suggest to remove the literature analysis parts from the manuscript.
    2. The data portal, implemented through cellxgene, is accessible for non-programmers to use. However, it is very easy to end up with an "Unexpected HTTP response 400, BAD REQUEST" error, with essentially no description of the cause of the error or how to rectify it. When this occurs (and in my experience it occurs very frequently), this also forces the user to refresh the page entirely, losing any progress they may have made. I see that the authors describe this error in their FAQ page, but their answer is not very intuitive and I was unsure of what they meant: "This happens because the samples you selected doesn't contain all "Group by" you want compare for each "Split by" group. You could confirm using the "Diff. groups" buttons.".

    I would therefore ask that the authors provide more detailed tutorials (ideally step-by-step) on common analyses that users will want to perform, hopefully minimizing the amount of frustration that users will encounter.

    1. Selection of samples is not very quick or intuitive. If I wanted to select only the samples from one specific GEO accession, I had to resort to individually checking the boxes of the sample IDs that I wanted. If I instead selected the GEO accession under the samples source ID, then used the "Subset to currently selected samples" button, I invariable got the HTTP error 400 message. Of course, this may simply reflect my lack of familiarity with cellxgene; I would nevertheless encourage the authors to improve the FAQ to include a step-by-step example for how to do common analyses/procedures.
    2. The second case study, centered on coagulation genes, is misguided. Alteration of coagulation lab values in severe COVID-19 patients is reflecting the general inflammatory state of these patients, and would not be expected to manifest on the transcriptional level in infected cells/tissues. Coagulation labs are measuring the functional status of the coagulation cascade, which is far-removed from the direct transcription of the corresponding genes - proteolytic processing of clotting factors, etc. As with CRP (see above comment), most clotting factors are transcribed almost exclusively in the liver (check GTEx portal); I would not expect upregulation of coagulation factors in lung cell lines/organoids/cultures etc after infection with SARS-CoV-2. I would recommend the authors to pick a different gene ontology set for a case study, as the current one focusing on coagulation is confusing in a pathophysiologic sense.
    3. The two large clusters of blood-derived samples vs other tissues is not surprising and the authors' interpretation is confusing. The authors write that "the COVID-19 signature was not able to overcome the tissue specificity and that immune cells might respond to SARS-CoV-2 differently." This should be immediately obvious given the pathophysiology of COVID-19 infection; the cell types that are directly infected by SARS-CoV-2 will of course have a distinct response compared to the circulating blood cells of COVID-19 patients, which are responding by mounting an immune response. There is no reason to expect a priori that the DEGs in the directly infected lung cells would be similar to that of immune cells that are mounting a response against the virus.
    4. The authors devote considerable space in the manuscript to exploring "batch effects" and trying to minimize them (pg10-11 Fig 4A-D, Fig S4). However, given that the compiled datasets are from entirely different experimental and biological systems (e.g. in vitro infection vs patient infection, different cell lines, timepoints after virus exposure, diverse tissues, varying disease severity), it is inappropriate to simply refer to all of these differences as "batch effects" alone. Usually, the term "batch effect" would refer to the same biological experiment/system (i.e. A549 cells infected with CoV vs control), but performed on different days or by different lab personnel - in other words, batch effects are purely due to technical differences. This term clearly does not apply when comparing samples from entirely different cell lines, or tissues, etc, and the authors should not keep describing these differences as batch effects that should be "corrected" out.

    Indeed, the authors themselves state that the main point of their "batch effect correction" efforts is only for PCA visualization. I therefore feel this section contributes very little to the overall manuscript, especially given the authors' own recommendation that all analyses should be performed on individual datasets (which I certainly agree with). I assume that the authors were required to provide some sort of dimensional reduction projection for the cellxgene browser, but this is more a quirk in their choice of platform for the web portal. Thus, this section of the manuscript should be deemphasized.

    1. Given the limitations of any combined multi-dataset analyses, one very useful feature would be to conduct "meta-analyses" across multiple datasets. For instance, it would be informative to find which genes are commonly DEGs in user-selected comparisons, calculated separately for each dataset and then cross-referenced across the relevant/user-selected datasets.

    Minor comments:

    1. Fig S1G, color legend should be added (I understand that these colors are the same from S1H).
    2. Mouseover text for trackPlot on the data portal is incorrect (it says the heatmap text instead).
    3. Abstract should be revised to describe only the 1093 final remaining RNA-seq samples after filtering/QC steps.
    4. Text in many figures is too small to be legible. I would suggest pt 6 font minimum for all figure text, including the various statistics in the figure panels.
    5. Are the DE analyses in Fig 1F specifically limited to control vs SARS-CoV-2/COVID-19 comparisons? Many of the samples included in this study are from other respiratory infections (labeled "other" in Fig 1B).
    6. The word cloud format is not conducive for understanding or interpretation. It would be much more informative to simply have a barplot or similar to clearly indicate the relative "abnudance" of a given gene among all 315 DE analyses.
    7. Claims of increased/decreased dataset separability should have statistical analysis on the silhouette score boxplots (Fig S4G-I).
    8. Regarding Fig 4E-F - what are the key genes that contribute to PC1, and how do they relate to the DEGs in Fig 4G?
    9. Statistics describing the relation between OASL And TNF/PPARGC1A should be included to justify the author's statements. This could be correlation, mutual information, regression, etc.
    10. There are several studies now that have performed scRNA-seq on the lung resident and peripheral immune cells of COVID-19 patients. To more definitively tie in their analyses in Fig 4J-K/Fig S5D-E (to affirm "its important role in the innate immune response in lungs"), the authors should assess whether OASL is upregulated in the lung macrophages of COVID-19 patients vs controls.
    11. The visualization and analysis functions in the data portal appear to work reasonably well out of the box. However, the download buttons for plots did not work in my hands. I realized that a workaround is to right click -> "Save image as" (which then downloads a .svg file), but this is not ideal and should be fixed to improve usability. I had tested the data portal on both Firefox and Edge browsers, using a Windows 10 PC.

    Significance

    The data portal appears to have useful analysis and visualization features, and the data collection appears to be quite comprehensive. I would strongly encourage the authors to continue collecting datasets as they become available and further improving the usability of the portal. As noted in the above comments, I think there is potential for their cellxgene-based browser to be useful to non-computational biologists, but at present, the data portal is not as simple to use as it should be. With further efforts to developing step-by-step tutorials for common analysis/visualization tasks, more informative case studies, and the other revisions suggested above, this study could be a valuable resource for the community. Of note, this review is written from the perspective of a primary wet-lab biologist with extensive bioinformatics experience but limited web development expertise.

    Referee Cross-commenting

    I agree with the comments of the other reviewers.

  4. Review Commons

    Note: This preprint has been reviewed by subject experts for Review Commons. Content has not been altered except for formatting.

    Learn more at Review Commons

    Referee #1

    Evidence, reproducibility and clarity

    Summary:

    The manuscript submitted by Djekidel et al entitled: "CovidExpress: an interactive portal for intuitive investigation on SARS-CoV-2 related transcriptomes" reports on a new web portal to search and analyze RNAseq data related to SARS-CoV-2 infections. The authors downloaded and reprocessed data of more than 40 different studies, which is available on the web portal along with all available meta data. The web portal allows to perform numerous differential expression and gene set enrichment analyses on the data and provides publication ready figures. Because of batch effects that could not be removed, the authors do not recommend to analyze data across studies at this point. The authors conclude that the web portal is unique and will allow scientists to rapidly analyze gene expression signatures related to SARS-CoV-2 infections with the potential to make new discoveries.

    Major comments:

    Based on the scientific literature, the web portal seems to be an unprecedented resource to search and analyze SARS-CoV-2-related RNAseq data and as such would certainly be a useful resource for the SARS-CoV-2 scientific community. The authors argue that new discoveries are possible by using their web portal in providing use cases. However, the section detailing the analyses the authors did to generate new hypotheses about genes potentially relevant in SARS-CoV-2 infections are very difficult to follow and without more guidance very difficult to reproduce with the web portal. It would require substantial expert knowledge in RNAseq data analysis without more information being provided. It also seems that key candidate genes identified by their analyses have all been studied or identified to be related to SARS-CoV-2 infections, so it is somewhat unclear whether new hypotheses can be generated by the reanalysis of RNAseq datasets, especially because combining the data from different studies is currently not recommended by the authors. The manuscript would benefit from providing fewer use cases but for each of them providing more information on how the portal and which studies were used to generate them and which findings were not described in the publication of the used studies. Some observations in the manuscript are not substantiated with significance calculations (see below). At times, the English writing (grammar) should be improved.

    Minor comments:

    Page 6 last sentence: The statement of this sentence is very much what one would expect. It remains unclear whether the authors mean this as a result to validate the processing of the RNAseq data or as a new discovery. Please, clarify.

    Figure 3A: The violin plots are so tiny that it is impossible to see any trends. It is also difficult to understand which categories one should compare with each other. If there is anything significant to observe, please, add a statistical test and better guide the reader.

    Figure 3C: A legend for the color scale is missing. The signal (I guess expression amounts) for SESN2 seems very weak and the same between ICU and non-ICU samples. What is the significance for assigning this gene to the group of genes being upregulated in ICU samples? Also contrary to what the authors state on page 8, SESN2 does not seem to be highly expressed in ICU samples, however, without knowing what the colors represent (fold changes or absolute expression values?) this is somewhat speculative.

    Page 9 first sentence: Please, specify what you mean by "starting list". Furthermore, in this paragraph, how do your results compare to the results from the study that you re-analyze here?

    Figure 3F: Please add labels to your axes and is there a particular reason why in a correlation plot like this one, the y and x axis are not shown with the same range and why does the y axis not start at 0?

    Page 9 second last sentence: It remains unclear which kind of analysis the authors intend to do here and what the starting question is. Please, try to rewrite with less technical terms (i.e. what do you mean by "precalculated contrasts"). In line with this, it remains unclear what Figure 3I is supposed to show. Please, provide some more information to readers who are not RNAseq analysis experts.

    Figure 3J is somewhat confusing. Why is the mean expression range indicated from 0 to 1 and why are all genes apparently having a mean expression of 1? Page 10 line 5-6. Are you referring to coagulation markers here or general expression patterns? In case of the latter, how does this statement fit to the paragraph about analyzing expression patterns of coagulation markers? Please, specify. And in line with this, are the highlighted genes in Figure 3K coagulation markers? If not, what is the relevance of these to make the point that one can use the portal to investigate the role of coagulation markers in SARS-CoV-2 infections?

    The appearance of describing batch effects and attempts to remove them from the studies was somewhat surprising on page 10 as I would expect this kind of results rather earlier in the results section before describing use cases of the data. You may consider changing the order of your results for a better flow. Page 11, second paragraph. Please, explain briefly what the silhouette score is supposed to reflect and thus how Figure S4G should be interpreted. The difference of both bars in Figure S4G is very marginal and thus, does not seem to support the statement of the authors that the ssGSEA scores-based projection is better unless you perform a significance test or I misunderstood. Please, clarify.

    Page 11, third paragraph: Figure 4B, to the best of my understanding, does not support the claim that samples clustered less according to study cohorts using the ssGSEA approach. Please, quantify the effect and test for significance or better explain.

    For the analyses described starting on page 12 it remains largely unclear whether they were conducted across studies or within studies and which studies were used. This section until the end of the results would especially benefit from providing more information on how the analyses were performed, either in the results or in the methods section.

    Figures 4J and 4K miss axis labels and since we look at correlations, the figures could be redrawn using the same ranges on x and y axis.

    Page 14 line 5: Is this the right figure reference here to Figure 4G? If yes, then it is unclear how Figure 4G supports the statement in this sentence. Please, clarify. Figure 2 is of too low resolution. Many details cannot be read. Please, provide a higher resolution figure.

    Significance

    Providing a single platform for the analysis of SARS-CoV-2-related RNAseq data is certainly of high value to the scientific community. However, as the portal and manuscript are currently presented, for scientists that are not RNAseq analysis specialists, more guidance would be required to understand and use correctly the functionalities of the portal. Unfortunately, because batch effects could not be removed from the studies, the authors, correctly, do not recommend to combine data from different studies for analyses, however, this likely will also limit the potential of the resource to make new discoveries beyond what the original studies have already published. As indicated above, the authors could support their claim by comparing their findings with findings published from the studies they reanalyzed. The portal is only of use to scientists studying SARS-CoV-2. I am not an expert in RNAseq data analysis and thus cannot comment on the technicalities, especially the processing of the RNAseq datasets.

  5. Version published to 10.1101/2021.05.14.444026v3 on bioRxiv
  6. Version published to 10.1101/2021.05.14.444026v2 on bioRxiv
  7. ScreenIT

    SciScore for 10.1101/2021.05.14.444026: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    NIH rigor criteria are not applicable to paper type.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    RNA-seq data analysis: Sequencing reads were quality filtered using TrimGalore (available on-line at https://www.bioinformatics.babraham.ac.uk/projects/trim_galore/).
    https://www.bioinformatics.babraham.ac.uk/projects/trim_galore/
    suggested: (Trim Galore, RRID:SCR_011847)
    Filtered reads were aligned to the human reference genome GRCh38.p12 using STAR (74), assuming that the RNA-seq experiment is strand-specific.
    STAR
    suggested: (STAR, RRID:SCR_004463)
    Next, MarkDuplicates from GATK (75) and CollectRnaSeqMetrics from Picard (available on-line at http://broadinstitute.github.io/picard/) were used to mark duplicated reads and compute mapping statistics.
    GATK
    suggested: (GATK, RRID:SCR_001876)
    Picard
    suggested: (Picard, RRID:SCR_006525)
    RSEM (76) was used to quantify read counts per gene based on Gencode v31 reference gene annotation, and expression values were converted to Fragments Per Kilobase of transcript, per Million mapped reads (FPKM) unit.
    Gencode
    suggested: (GENCODE, RRID:SCR_014966)
    The exception from the approach to assign the same strand specificity toward all samples from the same experiment was made for GSE147507 experiment, in which case the manual examination revealed that strand specificity status called by RSeQC was consistent within subseries of the samples, as emphasized by the sample names.
    RSeQC
    suggested: (RSeQC, RRID:SCR_005275)
    GSEApy was run with 1000 permutations and gene set size thresholds were set to 5 and 5000 for minimal and maximum size, respectively.
    GSEApy
    suggested: None
    The Seurat object was then converted into the .h5ad format using the SeuratDisk R package (https://github.com/mojaveazure/seurat-disk).
    SeuratDisk
    suggested: None

    Results from OddPub: Thank you for sharing your code and data.

    Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    Results from scite Reference Check: We found no unreliable references.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  8. Version published to 10.1101/2021.05.14.444026v1 on bioRxiv