Development and Evaluation of an Offline Vision–Language Model System for Estimating Nutrients from Meal Photographs on Personal Devices: The PLANS (Personal-device Local Assessment of Nutrients System) Study
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Background
Estimating nutrient intake from meal photographs with vision–language models (VLMs) is now practical, but patient photographs often cannot leave the institution. We developed PLANS (Personal-device Local Assessment of Nutrients System), in which a small open VLM runs entirely on a personal device.
Methods
Open VLMs were compared on two image sets with reference values (NutriImage, 887 Japanese meals; SNAPMe, 1,134 US meals), and the selected model (Qwen3-VL-4B, 4-bit) was run on an iPad Air without network access. Energy (primary outcome), protein, fat, carbohydrate and salt were evaluated by median absolute percentage error (MedAPE) and Spearman’s ρ. The effect of supplying the food weight, including errors in it, was examined.
Results
Energy estimates on the iPad were identical to those of a desktop-computer run for 84–86% of images (other values, 75–87%). MedAPE on the iPad was 37.0% and 36.5% for energy (26–28% of images within ±20%), 43–65% for the macronutrients and 68–101% for salt (ρ 0.51–0.85). On the desktop computer, supplying the true weight reduced MedAPE of the 4B model on NutriImage from 37.0% to 32.5% for energy and from 100.0% to 66.8% for salt; energy MedAPE stayed below the base value for declared weights of 0.5–1.25 times the true weight or rounded to 50 g, but rose markedly when the container weight was included.
Conclusions
An offline VLM on a tablet reproduced server-side energy accuracy and is a candidate tool for dietary counselling where photographs cannot leave the institution; given the wide limits of agreement, its usefulness in practice remains to be examined.
