DNA-LLM2Vec: Architecture-Aware Task-Conditioned Adaptation of Decoder DNA Foundation Models
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Decoder DNA foundation models pretrain with dense next-token prediction and scale to long sequences, but their representations are directional. This is a poor fit for genomic tasks that need sequence context both upstream and downstream of a locus. Bidirectional masked-language-model encoders supply this context, but they pretrain more sparsely and scale less easily to long windows. We introduce DNA-LLM2Vec, a staged decoder-to-encoder framework that opens bidirectional context, repairs the resulting mismatch with masked next-token prediction, and applies contrastive views encoding strand and positional invariance. We instantiate it on two architecturally distinct backbones, DNAGPT and HyenaDNA, and evaluate seven biological task families (grouped from GB, NT, GUE, and GUE+ EPI) with frozen linear probing. The benefits of conversion are architecture-aware and task-conditioned rather than uniform. Crop-based positional contrastive learning helps most for the extended-context families (enhancer–promoter interaction and Virus-COVID), raising frozen Avg-EPI MCC to 27.5% for DNAGPT and 26.7% for HyenaDNA, whereas for shorter, local families the results depend on backbone and tokenisation. Beyond classification, bidirectional context and MNTP improve causal-eQTL variant-effect prediction specifically at distance, and a base-pair mutagenesis diagnostic shows that adaptation organises motif-aware contact structure while reverse-complement training imprints strand-consistent geometry, yet neither predicts the best downstream view. DNA-LLM2Vec therefore produces frozen bidirectional representations whose most effective adaptation view depends on the task structure and the backbone. Code and checkpoints are available at semi-nothing/dna-llm2vec .
Key Points
-
DNA-LLM2Vec converts pretrained causal DNA foundation models into frozen, bidirectional encoders via bidirectional patching, masked next-token prediction (MNTP) repair, and contrastive adaptation.
-
Frozen linear probing (our primary protocol, reusable representation quality) and full fine-tuning (a supplementary analysis, downstream adaptation potential) are not interchangeable. Full fine-tuning reshapes rather than preserves the frozen task-family response, in an architecture-dependent way.
-
A base-pair mutagenesis motif-contact analysis provides an orthogonal, non-classification view: adaptation strengthens motif-aware regulatory structure, but this peaks at different adaptation stages across the two architectures and does not, by itself, predict the best downstream variant.
-
Ablations and controls support the design: the pretrained backbone supports the broad (Fam-mean) gains, not the objectives alone; the staged pipeline slightly outperforms its shortcuts; and results are stable under alternative pooling.
