A Selection Framework for Distilled AI Models in IoT-based Edge Applications

The rapid expansion of artificial intelligence (AI) in edge and Internet of Things (IoT) applications calls for the efficient deployment of lightweight AI models suited to resource-constrained edge applications. Knowledge distillation has become a key technique for compressing large "teacher" models into smaller "student" models, yet the absence of standardized guidelines for model selection hampers optimal deployment. This paper introduces a systematic framework for selecting distilled AI models in IoT-based edge systems, balancing accuracy, computational efficiency, and hardware constraints. The framework considers essential factors such as model types (predictive vs. generative), performance metrics (e.g., confusion matrix-based evaluation, latency), and hardware-specific requirements (e.g., FLOPs, memory footprint, and memory access cost). By advocating for standardized communication between AI model developers and IoT hardware manufacturers, it promotes transparency in system requirements and performance benchmarks. This work aims to contribute toward the development of foundational industry guidelines, helping developers navigate the evolving landscape of distilled AI models while addressing real-world challenges in edge computing, including real-time processing, energy efficiency, and data privacy, with the aim of encouraging growth in the distilled AI model community, inspired by how standardization fueled adoption in the early PC industry.