Learning Force Distribution Estimation for the GelSight Mini Optical Tactile Sensor Based on Finite Element Analysis

Contact-rich manipulation remains a major challenge in robotics. Optical tactile sensors like GelSight Mini offer a low-cost solution for contact sensing by capturing soft-body deformations of the silicone gel. However, accurately inferring shear and normal force distributions from these gel deformations has yet to be fully addressed. In this work, we introduce FEATS -- Finite Element Analysis for Tactile Sensing -- a machine learning approach using a U-net architecture to predict force distributions directly from the sensor's raw images. Our model, trained on force distributions inferred from Finite Element Analysis, demonstrates promising accuracy in predicting normal and shear force distributions. It also shows potential for generalization across sensors of the same type and for enabling real-time application.

Paper

Latest version: feats.pdf
Submitted to IEEE International Conference on Robotics & Automation (ICRA) 2025 (under review)

Code and Dataset

FEATS Dataset

Team

Erik Helmut^*1

Luca Dziarski^*2

Niklas Funk²

Boris Belousov³

Jan Peters^2,3,4,5

^*Authors contributed equally  ¹Department of Computational Engineering, Technical University of Darmstadt  ²Department of Computer Science, Technical University of Darmstadt  ³German Research Center for AI (DFKI)  ⁴Centre for Cognitive Science, Technical University of Darmstadt  ⁵Hessian Center for Artificial Intelligence (Hessian.AI), Darmstadt

BibTeX

@misc{FEATS2024,
    author       = {Helmut, Erik and Dziarski, Luca and Funk, Niklas and Belousov, Boris and Peters, Jan},
    title        = {Learning Force Distribution Estimation for the GelSight Mini Optical Tactile Sensor Based on Finite Element Analysis},
    howpublished = {Submitted to IEEE International Conference on Robotics and Automation (ICRA) 2025},
    note         = {Under review},
    year         = {2024},
}