Proprioception, often referred to as the “sixth sense”, is the ability through motor neurons to perceive and recognize the spatial posture of limbs, including the sense of joint position and movement and the perception of muscle force and extent, achieved through the coordination of the musculoskeletal system and various sensory organs. Developing the “sixth sense” technology for intelligent machines is a significant challenge in creating a new generation of smart robots. It requires mimicking human fingers’ adaptability during object gripping, joint position sensing, and precise force tactile perception during contact, while offering cost-effective and reliable flexible intelligent interaction methods in practical applications.
Recently, a research team led by Assistant Professor Chaoyang Song from the Department of Mechanical and Energy Engineering and Assistant Professor Fang Wan from the School of Design at the Southern University of Science and Technology (SUSTech) has published independent research titled “Proprioceptive Learning with Soft Polyhedral Networks” in the International Journal of Robotics Research. This work systematically introduces a design method for soft polyhedral network structures that can adaptively interact in all directions, and pioneered proprioceptive learning capabilities in robotics using in-finger vision. Their research uncovered the influence mechanisms of dynamic viscoelastic mechanics in soft robotic metamaterials on the learning of refined tactile perception under large deformations, enabling simultaneous 6D force-torque sensing and real-time 3D reconstruction. This technology can be applied to soft robotic tactile grasping, intelligent human-machine interaction, and touch-based reconstruction of object morphology, maintaining reliable visuo-tactile proprioceptive performance even after a million cycles of use.
In earlier research, the team independently invented a universal design method for a class of Soft Polyhedral Networks capable of omni-directional adaptation under large deformation. This structure provides cost-effective, high-performance, versatile, and reliable omni-directional adaptive coverage in unstructured physical interactions and has obtained multiple national and U.S. patents.
Figure 1. Soft Polyhedral Network design with in-finger vision
Based on this, they further proposed a miniature motion capture system design for tactile sensing using in-finger vision, achieving high-frame-rate unobstructed full-directional motion capture of the entire network structure during large deformations. Through extensive finite element simulation data and experimental calibration, they achieved real-time deformation simulation to reality (Sim2Real) learning, providing high-performance real-time proprioceptive sensing of joint positions and movements in the soft polyhedral network structure.
Figure 2. Learning adaptive kinesthesia with Sim2Real proprioception
Further focusing on the force perception during unstructured physical contact, the group’s research delves into the dynamic viscoelastic mechanisms of this type of soft metamaterial network structure, employing visuo-tactile fusion for precise tactile characterization and adaptive interaction testing. This work demonstrated reliable tactile perception performance even after a million use cycles, significantly enhancing the prediction accuracy of six-dimensional interactive forces and torques in soft robotic tactile interaction. This paves the way for accurately estimating multiple force perception and interaction performance parameters, including contact friction force.
Figure 3. Visual force learning for viscoelastic proprioception
Through this comprehensive study on flexible proprioception and fine force perception, the researchers proposed a learning method based on the soft polyhedral network structure, effectively endowing robots with a “sixth sense” at their fingertips. Through various system experiments, the team demonstrated that the soft polyhedral network structure, equipped with in-finger vision, combines cost-effectiveness, high performance, versatility, and reliability. It is capable of handling challenging unstructured human-machine interaction tasks, including precise prediction of contact friction force, tactile reconstruction of object geometry under large deformation, superior force tactile perception, and non-damaging gripping performance, as well as adaptive force tactile interaction performance comparable to human fingers.
Figure 4. Sensitive and robust grasping of the Soft Polyhedral Networks as robotic fingers
Combined with the generality of this patented design method, future expansions can cover a wide variety of applications, offering affordable and reliable force tactile human-machine interaction scenarios, especially in contexts like multimodal force tactile perception and learning for embodied intelligence, and precise force tactile manipulation of objects in underwater extreme environments, among various life, industrial, and unique scenarios.
Ph.D. students Xiaobo Liu and Xudong Han are the co-first authors of the paper, both from the Department of Mechanical and Energy Engineering. Professor Wei Hong from the Department of Mechanics and Aerospace Engineering is the co-author, and Assistant Professors Fang Wan and Chaoyang Song are the corresponding authors.
This research is supported by the National Natural Science Foundation of China Youth Fund Project, the National Natural Science Foundation of China Key Project, the Shenzhen Basic Research Special Key Project, the Shenzhen Key Laboratory of Intelligent Robots and Flexible Manufacturing, and the Guangdong Key Laboratory of Human Augmentation and Rehabilitation Robotics.
Paper link: https://doi.org/10.1177/02783649241238765