Stop Wasting Time Fine-Tuning: Traditional Classifiers Shine with LLM Embeddings for Political Textual Analysis

Large language models (LLMs) are widely applicable for political science research; however, many researchers do not have access to sufficient compute resources or data to fine-tune LLMs for their research purposes. As such, it is necessary to identify ways to deploy LLMs more efficiently for political science research applications. We examine LLMs for political science classification tasks and show that using LLMs as feature extractors for downstream classification models (an embed-then-classify pipeline) has performance comparable to or exceeding the performance of LLMs fine-tuned for classification (a fine-tune-then-classify pipeline), all while requiring less compute time and data. Furthermore, we demonstrate that both the embed-then-classify and fine-tune-then-classify pipelines significantly outperform zero-shot prompting for classification using decoder-only models, which is prevalent in the social sciences. We present a robust set of experiments with three decoder-only LLMs, 19 encoder-only LLMs, five classification models, and four fine-tuning strategies on a new political classification dataset. This dataset includes over 130,000 text sequences for multi-class classification and is made up of text extracted from a variety of government documents. We further validate our findings on two other political science textual datasets.