Systematic benchmarking of basecalling models for RNA modification detection with highly-multiplexed nanopore sequencing

Nanopore direct RNA sequencing (DRS) holds promise for advancing our understanding of the epitranscriptome by detecting RNA modifications in native RNA molecules. Recently, Oxford Nanopore Technologies (ONT) has released basecalling models capable of detecting several RNA modifications. However, their accuracy, sensitivity and specificity, as well as cross-reactivity against other modification types, remains largely unexplored. Here, we systematically benchmark modification-aware models by evaluating their performance on a highly-multiplexed panel of synthetic molecules covering all possible sequence contexts, as well as on biological samples from a diverse set of species. We find that modification-aware models reliably detect diverse RNA modification types across a broad range of sequence contexts. However, they are prone to elevated false positive rates and exhibit notable cross-reactivity with other RNA modification types. We show that the use of modification-free controls allows significant, yet incomplete removal of false positives, thus constituting an essential control. Finally, we demonstrate that basecalling error patterns and alterations in current features can identify differentially modified sites, for modifications for which modification-aware models are absent. Overall, our results underscore the utility and accuracy of modification-aware basecalling models for RNA modification detection, while highlighting the importance of including diverse control samples to mitigate false positive rates.