Visual-Tactile Based Manipulation Control for Complex Robot Manipulation Process via Geometric and Contact State Modeling

The accurate representation of object shape, pose, and tool-object contact states is of paramount importance in complex robot tool manipulation tasks. These facets are comprehensively modeled using tactile sensors, which provide a wealth of information. The tactile-based manipulation control framework presented here consists of three core components: the explorer, modeler, and controller. In the context of contact state modeling, we collect the moving trend between the grasped object and the visual-tactile sensor, which could be used for estimating the contact state. Addressing the challenge of precise object modeling, including both shape and pose, we employ the visual-tactile joint exploration and geometric modeling approach introduced in this study. The controller leverages the modeled object's shape and pose while estimating contact states through the contact state classifier. In scenarios where contact states exhibit uncertainty, the robot autonomously engages in exploration, returning to its established contact mode to successfully conclude the manipulation. Peg-hole-insertion experiments featuring pegs and holes of varying shapes have been conducted to empirically validate the efficacy of the presented manipulation control framework.