Managing Fuzziness: Leveraging LLMs for Discovering Credibility Indicators in Asylum Cases

In many areas of law, legal concepts are characterised by a having degree of fuzziness, which leads to “matters of degree” assessment when such concepts are applied in legal practice. In this paper, we examine how this fuzziness may be mapped and understood. We approach this problem through an abductive inspired discovery approach. Specifically, we show how LLMs can be employed to understand how decision-makers make inferences about applicants’ credibility in asylum cases and to capture indicators attached to such evidence assessments - an issue surrounded by significant disagreement in both legal scholarship and legal practice. We do this through a two-step methodology that recognizes the uncertainties surrounding LLMs and attempts to purposefully utilize the breadth of learned representations embedded in the model. Step one uses open-ended prompting to identify credibility indicators on a sample from our dataset; step two synthesizes these discoveries into generalizable categories and applies them to the full dataset. Our analysis demonstrates that LLMs can successfully identify credibility factors that correspond to established legal understanding of credibility’s role as evidence while revealing how these abstract principles translate to more a variety of credibility indicators in applied practice. Our scaled analysis across the full dataset reveals the relative frequency and importance of different credibility indicators in Danish asylum practice. This methodology advances both legal scholarship and computational social science by offering an approach to analyzing multi-variable and open-ended legal concepts at a greater scale. Our abductive inspired framework provides a systematic approach for empirically grounding theoretical understandings of fuzzy legal concepts, bridging the gap between abstract evidence law and actual decision-making practice in a way that is broadly applicable in law as well as other social science domains.