Contact-rich manipulation remains a major challenge in robotics. Optical tactile sensors like GelSight Mini offer a low-cost solution for contact sensing by capturing soft-body deformations of the silicone gel. However, accurately inferring shear and normal force distributions from these gel deformations has yet to be fully addressed. In this work, we introduce FEATS -- Finite Element Analysis for Tactile Sensing -- a machine learning approach using a U-net architecture to predict force distributions directly from the sensor's raw images. Our model, trained on force distributions inferred from Finite Element Analysis, demonstrates promising accuracy in predicting normal and shear force distributions for the commercially available GelSight Mini sensor. It also shows potential for generalization across indenters, sensors of the same type, and for enabling real-time application.
This video demonstrates a simple pick-and-place task using our FEATS model, which predicts force distributions from the GelSight Mini tactile sensor. The sensor is mounted on a gripper and is used during the task to capture tactile data in real time. The predicted force outputs from FEATS are compared against ground truth measurements from a Resense Hex-21 force/torque sensor operating at 100 Hz. In contrast, the FEATS model runs at 25 Hz, resulting in naturally slower prediction signals compared to the faster ground truth. Notably, the GelSight Mini sensor used in the demo was not part of the training dataset, highlighting the model's generalization capabilities. Throughout the demonstration, force prediction errors remain below 1 N, showcasing the accuracy and robustness of the FEATS model under real-world manipulation conditions.
Preprint: arXiv
Workshop Version:
OpenReview
Accepted at 2nd Workshop on Touch Processing: From Data to Knowledge (NeurIPS 2024)
*Authors contributed equally 1Department of Computational Engineering, Technical University of Darmstadt 2Department of Computer Science, Technical University of Darmstadt 3German Research Center for AI (DFKI) 4Centre for Cognitive Science, Technical University of Darmstadt 5Hessian Center for Artificial Intelligence (Hessian.AI), Darmstadt
@misc{helmut2025learningforcedistributionestimation,
title={Learning Force Distribution Estimation for the GelSight Mini Optical Tactile Sensor Based on Finite Element Analysis},
author={Erik Helmut and Luca Dziarski and Niklas Funk and Boris Belousov and Jan Peters},
year={2025},
eprint={2411.03315},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2411.03315},
}