FHIRBench: Benchmarking FHIR Clinical Data Serialization Strategies for Large Language Models

The integration of Large Language Models (LLMs) into clinical decision support systems requires transforming structured health data — primarily encoded in HL7 FHIR — into representations that maximize model comprehension. Despite growing adoption, no systematic benchmark examines how serialization strategy choice affects downstream clinical LLM performance, or how this interacts with model architecture and evaluation methodology.

We present FHIRBench, a controlled benchmark evaluating six FHIR serialization strategies across four frontier foundation models (Claude Sonnet 4.5, GPT-5.4, DeepSeek V3.2, Qwen3 32B) on three clinical tasks (question answering, reasoning, summarization) using 100 stratified synthetic FHIR R4 patient bundles. We employ a two-layer evaluation framework: Layer 1 (token-level F1) and Layer 2 (LLM-as-judge rubric on four clinical dimensions), yielding 7,200 evaluations per layer.

Our findings reveal four principal results. First, serialization significantly impacts quality on both evaluation layers, but the direction diverges: Condensed format outperforms Raw JSON on F1 for 3/4 models (Wilcoxon p < 10 ⁻¹⁷ , patient-level N = 100), while Raw JSON achieves higher judge scores for 3/4 models (p < 10 ⁻⁷ ). The Pareto analysis resolves this tension: Narrative achieves 95% of Raw JSON’s clinical quality at 83% fewer tokens. Second, model rankings completely reverse between layers — Claude ranks last on F1 but first on clinical quality (p = 1.0 × 10 ⁻⁶ ), demonstrating that single-metric evaluation produces misleading model selection. Third, a significant Model × Serializer interaction (Friedman p = 0.0009) precludes universal format recommendations, with GPT-5.4 favoring Raw JSON while Claude and open-weight models favor compressed formats. Fourth, Llama 3.1 70B exhibits 100% inference failure on complex patients despite operating within its nominal context window, revealing a patient-safety gap where AI assistance fails precisely for the patients who need it most.

These findings establish that clinical AI systems require (1) model-aware serialization middleware, (2) multi-layer evaluation frameworks, and (3) explicit capacity verification before deployment. FHIRBench — including all code, data, and evaluation harnesses — is publicly available at https://github.com/JacquelineChong/fhirbench . Benchmark data and raw results are archived at https://doi.org/10.5281/zenodo.21223883 .