Tactile Driven Reinforcement Learning Information Reconstruction for Enhanced Dexterous Hand Telemanipulation

Chenyang Miao1,3, Mingyu Sun2,3 Yingzhuo Jiang1,3, Yidong Chen4, Yunduan Cui*,1, Xinyu Wu1,
1Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China, 2College of Engineering, Southern University of Science and Technology, China 3University of Chinese Academy of Sciences, China 4Shenzhen ZHAOWEI Machinery & Electronics Co., Ltd, China *Cooresponding author: Yunduan Cui (e-mail: cuiyunduan@gmail.com)
Paper
Interpolation end reference image.

Tactile-Enhanced Dexterous Hand Telemanipulation (TEDHT)

A novel teleoperation framework for dexterous hands. It employs a hybrid reinforcement learning (RL) approach that integrates online and offline RL to efficiently learn and estimate the mapping between tactile signals from the dexterous hand and the pose information of the manipulated object, a critical yet often unobservable aspect of in-hand manipulation tasks.

Abstract

Controlling dexterous robotic hands to achieve human-level precision in complex in-hand manipulation tasks remains challenging due to high-dimensional joint control and unobservable physical interactions. While existing teleoperation methods focus on gesture retargeting, they often neglect critical tactile feedback during object manipulation. Inspired by human tactile perception and cerebellar control mechanisms, we propose Tactile-Enhanced Dexterous Hand Telemanipulation (TEDHT) – a hybrid reinforcement learning framework integrating online and offline RL. TEDHT addresses the critical challenge of unobservable object states in in-hand manipulation through three key contributions:

  1. Hybrid RL Framework: Combines online and offline reinforcement learning to establish the mapping between tactile signals and manipulated object poses.
  2. Tactile-Driven Compensation: Implements an action compensation mechanism that optimizes control using reconstructed physical variables.
  3. Comprehensive Validation: Demonstrates enhanced control performance and generalization capability through both simulation and real-world hardware experiments.

Our framework provides a new paradigm for integrating tactile feedback with teleoperation systems, significantly advancing dexterous manipulation capabilities.

Experiments

Testing performances of TEDHT in manipulating three different objects.

Testing performances of TEDHT in manipulating three different objects.

Snapshots in one testing episode in manipulating 1.5 Kg hammer.

Snapshots in one testing episode in manipulating 1.5 Kg hammer. TEDHT and vanilla teleoperation had the same input actions aTE from the glove.

Trajectories of the XYZ positions of the manipulated object and selected tactile and finger joint states.

Trajectories of the XYZ positions of the manipulated object and selected tactile and finger joint states using TEDHT and vanilla teleoperation in one testing episode under the same input actions aTE from the glove.

Testing performances of TEDHT in manipulating three different objects on the real-world DexHand.

Testing performances of TEDHT in manipulating three different objects on the real-world DexHand.

Snapshots in one testing episode with 497 g hammer.

Snapshots in one testing episode with 497 g hammer. Both TEDHT and vanilla telemanipulation added similar noise-like perturbations to their action aTE inputs from the glove.

Video