Variational Interpretable Framework for Multimodal Instruction Execution

Empowering agents with the ability to understand and follow complex language instructions in diverse environments is a crucial goal in both robotics and artificial intelligence. However, the substantial requirement of paired multimodal data, consisting of natural language commands and their corresponding trajectories, poses a significant challenge in real-world applications. In this work, we propose a novel generative learning framework, \textbf{IntraMIX} (Interpretable Multimodal Instruction eXecutor), tailored to semi-supervised instruction-following tasks. Our approach leverages a sequential multimodal generative mechanism to jointly encode and reconstruct both paired and unpaired data through shared latent representations. By extending traditional multimodal variational autoencoders into a sequential domain and introducing an attention-compatible latent structure, IntraMIX successfully addresses the limitations of prior models in sequence-to-sequence tasks. Moreover, we demonstrate how IntraMIX can be integrated into the prevalent speaker-follower pipeline by proposing a new regularization strategy that mitigates overfitting when leveraging unpaired trajectories. Experiments conducted in the BabyAI and Room-to-Room (R2R) environments confirm the effectiveness of our model, where IntraMIX improves instruction-following performance under limited supervision and enhances the speaker-follower framework by 2%–5%. Our results suggest that generative modeling presents a promising pathway toward more data-efficient instruction-following agents.