Effects of Dish Name, Visible Amount, Persona, and Explicit Instruction on Energy Estimates From Food Photographs by 26 Vision–Language Models: Comparative Evaluation Study
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Background
Vision–language models (VLMs) are being considered for estimating energy intake from meal photographs. Prompt descriptions such as a Japanese-dietitian persona shift the estimates, suggesting their use for calibration; what the estimates rest on was unknown.
Objective
We examined whether prompt descriptions can serve as calibration, at which training stage responses to a nationality persona and to an amount instruction arise, and whether estimates follow the visible amount or one associated with the identified dish.
Methods
Eleven commercial and 15 open-weight models were evaluated on NutriImage (887 Japanese single-dish photographs) and SNAPMe (1,134 US meal photographs) under eight prompt conditions (no description, five personas, and smaller- or larger-portion instructions). With one model, images were cropped or replaced by the dish name. Before-and after-meal photographs of 804 meals (ACETADA) were estimated. Pre-trained and post-trained weights of Gemma 3 and InternVL3.5 8B were compared by likelihood scoring. Prompts were compared with numerical calibration.
Results
Where the dish was named as in the full image, a 25% crop changed neither the median absolute percentage error (MedAPE; 33.8% vs 34.4%) nor the persona shift (−20 vs −20 kcal); dish names alone gave similar accuracy (MedAPE 31.7%); nearly empty trays still received 39%–75% of the before-meal estimate, often described as already eaten. Between models, the persona shift was related to baseline bias (r = −0.77) and less to instruction following (r = −0.36); the instruction shift showed the reverse (r = −0.14 and −0.95). In Gemma 3, the Japan–US contrast was 2–4 times larger before post-training, and the instruction response 2–4 times larger after it (InternVL3.5 8B: 3 and 1.4 times). Across images, the Japan–US contrast was uniform on US meals and, for Japanese dishes in the commercial models, Gemma 3, and Qwen3-VL 32B, larger where overestimated. In these models, the persona lowered MedAPE for Japanese dishes as much as, or slightly more than, subtracting the mean bias, and no more than dividing by a ratio; on US meals it did not; in open-weight families that responded little, only numerical calibration lowered error. The smaller-portion instruction raised MedAPE in models with little bias (eg, 30.0% to 50.2%).
Conclusions
In the model examined most closely, the estimates behaved as if anchored on an amount associated with the identified dish and only partly corrected by the visible amount. Prompt descriptions moved them by amounts depending on model and cuisine, not on the correction needed; a persona is therefore not proposed as a general means of calibration. In a facility, the findings suggest showing the identified dish for correction, calibrating by a ratio measured on reference meals with the deployed model, keeping amount statements out of the prompt, and not deriving intake from before– after-meal differences.
