Learning Force Distribution Estimation for the GelSight Mini Optical Tactile Sensor Based on Finite Element Analysis

Contact-rich manipulation remains a major challenge in robotics. Optical tactile sensors like GelSight Mini offer a low-cost solution for contact sensing by capturing soft-body deformations of the silicone gel. However, accurately inferring shear and normal force distributions from these gel deformations has yet to be fully addressed. In this work, we introduce FEATS -- Finite Element Analysis for Tactile Sensing -- a machine learning approach using a U-net architecture to predict force distributions directly from the sensor's raw images. Our model, trained on force distributions inferred from Finite Element Analysis, demonstrates promising accuracy in predicting normal and shear force distributions for the commercially available GelSight Mini sensor. It also shows potential for generalization across indenters, sensors of the same type, and for enabling real-time application.

Paper

Preprint: arXiv

Workshop Version: OpenReview
Accepted at 2nd Workshop on Touch Processing: From Data to Knowledge (NeurIPS 2024)

Code and Dataset

FEATS Dataset

Team

Erik Helmut^*1

Luca Dziarski^*2

Niklas Funk²

Boris Belousov³

Jan Peters^2,3,4,5

^*Authors contributed equally  ¹Department of Computational Engineering, Technical University of Darmstadt  ²Department of Computer Science, Technical University of Darmstadt  ³German Research Center for AI (DFKI)  ⁴Centre for Cognitive Science, Technical University of Darmstadt  ⁵Hessian Center for Artificial Intelligence (Hessian.AI), Darmstadt

BibTeX

@misc{helmut2024feats,
    title={Learning Force Distribution Estimation for the GelSight Mini Optical Tactile Sensor Based on Finite Element Analysis},
    author={Erik Helmut and Luca Dziarski and Niklas Funk and Boris Belousov and Jan Peters},
    year={2024},
    eprint={2411.03315},
    archivePrefix={arXiv},
    primaryClass={cs.RO},
    url={https://arxiv.org/abs/2411.03315},
}