Our research focuses on robot learning and control for dexterous manipulation, building robots that use touch and vision to handle objects skillfully and reliably in the physical world. Our projects are below, newest first.
A fingertip tactile sensor whose single camera either looks through its gel skin or reads contact, depending on the lighting, plus a model that fuses both views to estimate object pose and contact shape.
Lets a world model adapt to the physics of an unfamiliar scene, such as friction or mass, by watching a brief exploratory interaction and condensing it into a token that steers its predictions, with no fine-tuning.
Teaches a humanoid to hold and move bulky objects with its arms and torso, not just its hands, using a policy learned in simulation that transfers directly to a real Unitree G1.
A review of how touch is used in robot learning, from learned tactile representations and touch-aware policies to tactile foundation and world models, with open problems in transferring across sensors and robots.
A whole-body controller that lets a humanoid walk, carry, and push while bearing heavy loads, trained with reinforcement learning guided by a model of how the payload shifts the robot's balance.
Fixes simulation-trained dexterous hand policies that stumble on real hardware: a person physically guides only the struggling fingers, and those corrected runs, with fingertip force readings, train a stronger force-aware policy.
A reinforcement-learning recipe that pre-trains a dexterous arm-hand policy on general object reorientation in simulation, then fine-tunes it for skills like peg insertion, and transfers to two real robot hands without real-world training.
Moves touch-based policies from simulation to real tactile sensors without simulating those sensors, by training on contact signals the simulator already provides and then mapping real sensor readings into that learned space.
An open-source, camera-based tactile endcap for tensegrity robots that estimates full six-axis contact forces and torques from gel deformation and detects ground contact, built with a glue-free fabrication method so many units are cheap to make.
Gives vision-language-action policies a sense of touch by drawing tactile shear readings as arrows onto the camera images they already use, which improves contact-rich manipulation without changing the model or adding tactile pre-training.
A simulation framework in IsaacSim that procedurally builds photorealistic, building-scale indoor environments with articulated, physically simulated objects, plus a sim-to-real interface for running a Spot robot with an arm.
A benchmark of 16 long-horizon manipulation tasks that test whether generalist robot policies can remember earlier events, used to compare different ways of adding memory to vision-language-action models.
Transfers manipulation policies learned from human demonstrations with tactile gloves to a robot hand with different touch sensors, by learning a shared tactile representation without paired human and robot data.
A tactile simulator that models how a gel sensor surface stretches and slips under contact, so reinforcement learning policies trained in simulation on shear signals carry over directly to real GelSight sensors for insertion, packing, and drawer tasks.
Uses vibrating tensioned strings as a touch sensor. Contact microphones pick up how touching a string changes its sound, which lets a robot work out where and how hard it is being pressed and notice slipping.
Learns a representation of how manipulators move objects from large collections of unlabeled human videos, then fine-tunes it on a handful of robot demonstrations so new skills need much less robot data.
Estimates where a grasped object sits in the hand and where it touches the world at the same time, by combining tactile readings with physics constraints in a factor graph. This helps when the camera view is blocked or noisy.
Makes imitation-learned assembly policies more precise. It keeps a loosely held part on a small set of feasible poses and uses touch to correct small in-hand shifts before inserting pegs, threading nuts, or plugging in USB connectors.
Trains industrial robot arms to assemble timber mortise-and-tenon joints using diffusion policies guided by force-torque feedback. The policies keep working when the joint is moved up to a centimeter from where the robot expects it.
Shows that changing how often each task is drawn during training, and sampling rare tasks more as training goes on, helps robot policies learn from datasets dominated by a few common tasks, in simulation and on a real Franka arm.
Generates what one visuo-tactile sensor would feel from another sensor's readings, either with paired data or via an intermediate depth map, so tactile models trained on one sensor can be reused on a different one.
Hydrosoft is a computationally efficient, path-dependent, and differentiable model for simulating and controlling soft, compliant robotic components in dexterous manipulation tasks.
AimBot is a lightweight visual augmentation technique that provides explicit spatial cues (e.g., shooting lines and scope reticles) to improve any Vision-Language-Action (VLA) models.
ViTaSCOPE is a neural implicit representation that is trained entirely in simulation to integrate vision and touch for in-hand object pose and extrinsic contact estimation.
A novel approach to reposition object in free-space using a parallel jaw gripper, by utilizing the micro vibration at finger tips. We derive a sliding motion model for objects clamped in a parallel gripper with two symmetric, variable-friction contact patches, and propose a motion planner that coordinates end-effector finger trajectories and fingertip vibrations to achieve the desired object pose.
A robot picks exploratory motions that are most informative about where a hidden, movable object sits and how it is oriented, updating its belief from contact and vision until the object's pose is pinned down.
Combines a learned model of a soft tool with a contact optimization step to estimate how a rigid object moves and which forces act on it when a compliant tool pushes or pivots it.
A time-dependent flow model of a 3D-printing dispensing system for thick pastes, covering pump, mixer and nozzle, that predicts how extruded flow lags behind commands and how much material bulges at sharp corners.
Pairs a vision-language supervisor that describes mistakes and how to fix them with a language-conditioned manipulation policy, trained on demonstrations augmented with failure-and-recovery examples.
Uses a video generation model, guided by a short instruction plus pointing gestures, to imagine how a task should unfold, and then turns that generated video into robot actions.
Physics-informed neural networks that recover the unknown external source acting on a system, together with its full state, from partial and noisy observations.
A diffusion model that translates touch signals between sensors, so a manipulation method built for Soft Bubble sensors can run on a robot fitted with GelSlims.
An open-source, easier-to-build vision-based tactile finger with a perception library for depth and shear estimation, aimed at making tactile research reproducible.
A novel approach that leverages a generative model to reconstruct a local 3D representation of an object based solely on its tactile signature. This 3D reconstruction not only facilitates precise pose estimation but also allows for the quantification of uncertainty, providing a robust framework for tactile-based perception in robotics.
Using representations of tactile feedback to transfer force-rich compliant policies from high-end robots to those without force feedback. This work involves diffusion-based behavior cloning and vision-based tactile sensors.
Contrastive pretraining on paired touches from two different tactile sensors, producing a shared representation that lets touch models move from one sensor to the other without retraining.
Training neural implicit functions to reconstruct the high-dimensional raw tactile dataset. It produces compact representations that capture the underlying structure of the tactile sensory inputs. We demonstrate the efficacy of this representation on the downstream task of in-hand object pose estimation, achieving improved performance over image-based methods while simplifying downstream models.
Estimating contact locations during manipulation using a multimodal visual-auditory approach. We use contact microphones, speakers, and a static camera to detect interactions between grasped objects and their environment. Our method uniquely transfers auditory feedback into a simulated environment, enabling accurate zero-shot contact estimation in real-world scenes without additional training, as shown in different environments.
Using extended dual limit surfaces for bimanual in-hand manipulation. Our approach leverages two cooperative patch contacts on either side of the object to iteratively reposition it within the robot’s grasp by alternating between sliding and sticking motions. We demonstrate the effectiveness of our approach in simulation and real-world experiments, showing significant improvements in object stability and pose accuracy across various object geometries.
Contact-rich non-prehensile manipulation using grasped objects. Our approach solves for robot actions that drive object poses and forces to desired values while obeying the complex dynamics induced by the sensors as well as the constraints imposed by static equilibrium, object kinematics, and frictional contact.
Language-conditioned and contact-rich tool manipulation policy. We show a generalizable and transferable manipulation framework for controlling and planning interactions between tool and the environment.
Touch-based control for objects with geometric aliasing. Our method introduces a novel tactile benchmarking task: the tactile cartpole.
Jointly estimating how a tool deforms and contacts various shapes, both in simulation and on a physical robot system. Our proposed representation uses implicit neural fields to represent both the deforming tool geometry and contact interface between tool and environment.
In-hand object pose estimation with a bimanual robot system. The robots use multiple contact actions along with proprioception and tactile feedback to estimate the pose of each tool.
Mechanics and planning algorithms to slide an object on a horizontal planar surface via frictional patch contact made with its top surface.
Object manipulation and tool use with deformable high-resolution membrane sensors. The robot learns the dynamics of the deformable membrane use them to control the pose of grasped objects and transmitted forces to the environment.
Our novel approach for, and real-world demonstration of, multimodal visuotactile state-estimation and dynamics prediction for deformable objects.
Representing and controlling extrinsic contact between a compliant tool and the environment, including in the presence of obstacles.
Doing multi-modal reinforcement learning in simulated and real world environments with transformer networks.
In-hand object pose estimation. The robot uses only proprioceptive tactile feedback to estimate the pose of grasped objects by leveraging the contacts it makes between them.
Learning multimodal neural implicit representations capable of representing deformable objects!
Learning hierarchical multimodal representation learning for complex manipulation tasks.
Using tactile feedback to localize objects blindly in clutter. Our robot rummages, clusters objects it touches, and picks the desired object.
Using tactile feedback to blindly navigate novel environments and build trap-aware representations that help the robot achieve its goals.
Developing systems that are capable of autonomous picking and stowing as part of our entry to the Amazon Robotics Challenges.
Developing models that bring together the best of physics-based and data-driven methods.
Identification and evaluation of commonly used contact models for robotics applications.
Developing models and controllers that enable object manipulation with high-resolution and highly deformable tactile sensors.