The potential of short-wave infrared hyperspectral imaging and deep learning for dietary assessment: a prototype on predicting closed sandwiches toppings

Accurate measurement of dietary intake without interfering in natural eating habits is a long-standing problem in nutritional epidemiology. We explore the applicability of hyperspectral imaging and machine learning for dietary assessment of home-prepared meals, by building a proof-of-concept which automatically detects food ingredients inside closed sandwiches. Individual spectra were selected from 24 hyperspectral images of assembled closed sandwiches, preprocessed with SNV-filtering, derivatives, and subsampling, and fed into a multilayer perceptron. The resulting models had an accuracy score of ~ 80% prediction of the type of bread, ~ 60% for predicting butter, and ~ 24% for filling type. Further analysis on non-homogeneous mixed food items, using computer vision techniques, will contribute towards a generalizable system. While there are still significant technical challenges to overcome before such a system can be routinely implemented in studies of free-living subjects, we believe it holds promise as a future tool for nutrition research and population intake monitoring.