DYNAMO: Vision-Initialized Physics-Based Dynamic Motion Prediction for Object Manipulation

This paper presents DYNAMO, a pipeline combining a novel training-free vision-based 6D pose estimation with nonsmooth physics simulation to enable manipulation of a cluster of rigid objects. We assess the possibility to predict the physical outcomes of dynamic actions using approximate object inertia, friction, and geometry, derived from visual and potentially tactile inspection. To allow the onboarding of unseen objects, the training-free 6D pose estimation and shape reconstruction framework leverages foundation models for object detection, adaptive retrieval of clustered templates, image feature matching, and nonlinear optimization. Experimental results validate the accuracy of the pose estimates on the YCB-V dataset and the predictive capabilities of the vision-initialized physics engine in a dual-arm manipulation scenario. DYNAMO is envisioned to complement simulation-based plan-and-control frameworks and is seen as a first step towards robust and explainable real2simsim2real approaches for the manipulation of a cluster of objects in dynamic settings. Our source code is available at https://gitlab.tue.nl/20220548/dynamo.