A synthetic data generation framework for scalable and resource-efficient medical AI assistants

Large language models (LLMs) show promise in the medical field but remain limited by high computational costs and privacy needs. We present SCALEMED (Scalable Clinical Assistants and LEarning for MEDicine), a data centric framework designed to train and use specialized medical models on standard hardware. AnnotatorMed, an open source annotation tool supporting local data labelling, is also developed thereby reinforcing privacy. For dermatology, a visually demanding field, SCALEMED integrates open-access (OA) data, PubMed (PM) literature, and instruction-following (IF) to create DermaSynth, a comprehensive dataset of dermatological images and reports via knowledge transfer with 1.2 million samples. We demonstrate DermatoLlama, a vision-language model that inherits high-level reasoning using DermaSynth yet remains resource-efficient for local deployment. By unifying open-source tools, privacy-focused design, and modular training approaches, SCALEMED provides adaptable AI solutions in resource-limited clinical settings, such as our exemplar of dermatology. This work highlights a practical path for healthcare institutions to adopt LLM-driven decision support systems without the burdens of large-scale architectures or external data exposure by reaching the success of state-of-the-art (SOTA) vision LLMs.