Summary

Learning representations in the joint domain of vision and touch can improve manipulation dexterity, robustness, and sample-complexity by exploiting mutual information and complementary cues. Here, we present Visuo-Tactile Transformers (VTTs), a novel multimodal representation learning approach suited for model-based reinforcement learning and planning. Our approach extends the Visual Transformer to handle visuo-tactile feedback. Specifically, VTT uses tactile feedback together with self and cross-modal attention to build latent heatmap representations that focus attention on important task features in the visual domain. We demonstrate the efficacy of VTT for representation learning with a comparative evaluation against baselines on four simulated robot tasks and one real world block pushing task. We conduct an ablation study over the components of VTT to highlight the importance of cross-modality in representation learning.

Our proposed algorithm **Visual Tactile Transformers (VTTs)** enable effective real world multi-modal reinforcement learning. Here, the robot incorporates both visual and force/torque feedback to inform the policy.

Authors

Yizhou Chen

Andrea Sipos

Mark Van der Merwe

Nima Fazeli

Paper and Code

In 6th Conference on Robot Learning (CoRL 2022), Auckland, New Zealand.