Vision-based tactile measurement method and apparatus, chip, and storage medium
A vision-based tactile measurement method is provided, performed by a computer device (e.g., a chip) connected to a tactile sensor, the tactile sensor including a sensing face and an image sensing component, and the sensing face being provided with a marking pattern. The method includes: obtaining an image sequence collected by the image sensing component of the sensing face, each image of the image sequence comprising one instance of the marking pattern; calculating a difference feature of the marking patterns in adjacent images of the image sequence; and processing the difference feature of the marking patterns using a feedforward neural network to obtain a tactile measurement result, a quantity of hidden layers in the feedforward neural network being less than a threshold.
1. A vision-based tactile measurement method performed by a computer device connected to a tactile sensor, the tactile sensor comprising a sensing face and an image sensing component, and the sensing face being provided with a marking pattern; and the method comprising:
obtaining an image sequence collected by the image sensing component of the sensing face that is in physical contact with a surface of an object, each image of the image sequence comprising one instance of the marking pattern;
calculating a difference feature of the marking patterns in adjacent images of the image sequence; and
processing the difference feature of the marking patterns using a feedforward neural network to obtain a tactile measurement result, further comprising:
calling a hidden layer in the feedforward neural network to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the surface of the object, the feature presentation further including a curvature prediction on the surface of the object; and
calling an output layer in the feedforward neural network to process the feature representation to obtain the tactile measurement result, the tactile measurement result comprising a local curvature radius of the surface of the object.
2. The method according to claim 1 , wherein n hidden neurons are provided in the hidden layer, n being an integer;
the hidden layer is constructed based on hidden neurons of a logistic sigmoid function; and
the output layer is constructed based on output neurons of a normalized exponential softmax function or linear output neurons.
3. The method according to claim 1 , wherein the feedforward neural network comprises: a location estimation model, and the location estimation model comprises a first hidden layer and a first output layer;
the calling the hidden layer in the feedforward neural network to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the surface of the object comprises:
calling the first hidden layer in the location estimation model to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of a contact location on the surface of the object; and
the calling the output layer in the feedforward neural network to process the feature representation to obtain the tactile measurement result comprises:
calling the first output layer in the location estimation model to process the feature representation of the contact location, to obtain the contact location on the surface of the object.
4. The method according to claim 1 , wherein the feedforward neural network comprises: a contact force estimation model, and the contact force estimation model comprises a second hidden layer and a second output layer;
the calling the hidden layer in the feedforward neural network to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the surface of the object comprises:
calling the second hidden layer in the contact force estimation model to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the contact force on the surface of the object; and
the calling the output layer in the feedforward neural network to process the feature representation to obtain the tactile measurement result comprises:
calling the second output layer in the contact force estimation model to process the feature representation of the contact force, to obtain three-dimensional information of the contact force on the surface of the object, the three-dimensional information comprising at least one of a magnitude and a direction.
5. The method according to claim 1 , wherein the feedforward neural network comprises: a surface classification model and at least two curvature estimation models; and
the processing the difference feature of the marking patterns using a feedforward neural network to obtain a tactile measurement result comprises:
calling the surface classification model to perform surface recognition on the difference feature of the marking patterns to obtain a surface type of the surface of the object; and
calling a target curvature estimation model in the at least two curvature estimation models based on the surface type to perform the curvature prediction on the surface of the object, to obtain the local curvature radius of the surface of the object.
6. The method according to claim 5 , wherein the curvature estimation model comprises: a spherical surface estimation model and a cylindrical surface estimation model; and the calling a target curvature estimation model in the at least two curvature estimation models based on the surface type to perform the curvature prediction on the surface of the object, to obtain the local curvature radius of the surface of the object comprises:
in a case that the surface type is a spherical surface, calling the spherical surface estimation model to perform first curvature prediction on the spherical surface, to obtain the local curvature radius of the spherical surface; and
in a case that the surface type is a cylindrical surface, calling the cylindrical surface estimation model to perform second curvature prediction on the cylindrical surface, to obtain the local curvature radius of the cylindrical surface.
7. The method according to claim 5 , wherein the surface classification model comprises a third hidden layer and a third output layer; and the calling the surface classification model to perform surface recognition on the difference feature of the marking patterns to obtain a surface type of the surface of the object comprises:
calling the third hidden layer in the surface classification model to perform surface recognition on the difference feature of the marking patterns to obtain a feature representation of the surface type; and
calling the third output layer in the surface classification model to process the feature representation of the surface type, to obtain the surface type of the surface of the object.
8. The method according to claim 6 , wherein the spherical surface estimation model comprises a fourth hidden layer and a fourth output layer; and the calling the spherical surface estimation model to perform first curvature prediction on the spherical surface, to obtain the local curvature radius of the spherical surface comprises:
calling the fourth hidden layer in the spherical surface estimation model to perform the first curvature prediction on the spherical surface, to obtain a feature representation of the curvature prediction of the spherical surface; and
calling the fourth output layer in the spherical surface estimation model to process the feature representation of the curvature prediction of the spherical surface, to obtain the local curvature radius of the spherical surface.
9. The method according to claim 6 , wherein the cylindrical surface estimation model comprises a fifth hidden layer and a fifth output layer; and the calling the cylindrical surface estimation model to perform second curvature prediction on the cylindrical surface, to obtain the local curvature radius of the cylindrical surface comprises:
calling the fifth hidden layer in the cylindrical surface estimation model to perform the second curvature prediction on the cylindrical surface, to obtain a feature representation of the curvature prediction of the cylindrical surface; and
calling the fifth output layer in the cylindrical surface estimation model to process the feature representation of the curvature prediction of the cylindrical surface, to obtain the local curvature radius of the cylindrical surface.
10. The method according to claim 1 , wherein the marking pattern comprises at least two marking points, and the difference feature of the marking patterns comprises at least one of displacement and deformation of the marking points.
11. The method according to claim 1 , wherein the marking pattern comprises a grid, and the difference feature of the marking patterns comprises at least one of displacement of grid points and deformation of grid lines in the grid.
12. The method according to claim 1 , wherein the calculating a difference feature of the marking patterns based on the marking patterns in adjacent images of the image sequence comprises:
determining two closest marking patterns from adjacent images at an i th frame and an (i+1) th frame in the image sequence as the same marking pattern, i being an integer; and
calculating the difference feature of the marking patterns based on at least one of locations and deformation of the marking patterns in the images at the i th frame and the (i+1) th frame.
13. A computer device in connection with a tactile sensor, the tactile sensor comprising a sensing face and an image sensing component, and the sensing face being provided with a marking pattern, the computer device further comprising a memory and a processor, wherein the memory stores a plurality of computer-readable instructions; and when the processor executes the plurality of computer-readable instructions, causes the computer device to perform a plurality of operations including:
obtaining an image sequence collected by the image sensing component of the sensing face that is in physical contact with a surface of an object, each image of the image sequence comprising one instance of the marking pattern;
calculating a difference feature of the marking patterns in adjacent images of the image sequence; and
processing the difference feature of the marking patterns using a feedforward neural network to obtain a tactile measurement result, further comprising:
calling a hidden layer in the feedforward neural network to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the surface of the object, the feature presentation further including a curvature prediction on the surface of the object; and
calling an output layer in the feedforward neural network to process the feature representation to obtain the tactile measurement result, the tactile measurement result comprising a local curvature radius of the surface of the object.
14. The computer device according to claim 13 , wherein the marking pattern comprises a grid, and the difference feature of the marking patterns comprises at least one of displacement of grid points and deformation of grid lines in the grid.
15. The computer device according to claim 13 , wherein the calculating a difference feature of the marking patterns based on the marking patterns in adjacent images of the image sequence comprises:
determining two closest marking patterns from adjacent images at an i th frame and an (i+1) th frame in the image sequence as the same marking pattern, i being an integer; and
calculating the difference feature of the marking patterns based on at least one of locations and deformation of the marking patterns in the images at the i th frame and the (i+1) th frame.
16. The computer device according to claim 13 , wherein the feedforward neural network comprises: a location estimation model, and the location estimation model comprises a first hidden layer and a first output layer;
the calling the hidden layer in the feedforward neural network to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the surface of the object comprises:
calling the first hidden layer in the location estimation model to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of a contact location on the surface of the object; and
the calling the output layer in the feedforward neural network to process the feature representation to obtain the tactile measurement result comprises:
calling the first output layer in the location estimation model to process the feature representation of the contact location, to obtain the contact location on the surface of the object.
17. The computer device according to claim 13 , wherein the feedforward neural network comprises: a contact force estimation model, and the contact force estimation model comprises a second hidden layer and a second output layer;
the calling the hidden layer in the feedforward neural network to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the surface of the object comprises:
calling the second hidden layer in the contact force estimation model to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the contact force on the surface of the object; and
the calling the output layer in the feedforward neural network to process the feature representation to obtain the tactile measurement result comprises:
calling the second output layer in the contact force estimation model to process the feature representation of the contact force, to obtain three-dimensional information of the contact force on the surface of the object, the three-dimensional information comprising at least one of a magnitude and a direction.
18. A non-transitory computer-readable storage medium, storing a plurality of computer-readable instructions, wherein the plurality of computer-readable instruction when executed by a processor of a computer device in connection with a tactile sensor, the tactile sensor comprising a sensing face and an image sensing component, and the sensing face being provided with a marking pattern, cause the computer device to perform a plurality of operations including:
obtaining an image sequence collected by the image sensing component of the sensing face that is in physical contact with a surface of an object, each image of the image sequence comprising one instance of the marking pattern;
calculating a difference feature of the marking patterns in adjacent images of the image sequence; and
processing the difference feature of the marking patterns using a feedforward neural network to obtain a tactile measurement result, further comprising:
calling a hidden layer in the feedforward neural network to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the surface of the object, the feature presentation further including a curvature prediction on the surface of the object; and
calling an output layer in the feedforward neural network to process the feature representation to obtain the tactile measurement result, the tactile measurement result comprising a local curvature radius of the surface of the object.
19. The non-transitory computer-readable storage medium according to claim 18 , wherein the marking pattern comprises a grid, and the difference feature of the marking patterns comprises at least one of displacement of grid points and deformation of grid lines in the grid.
20. The non-transitory computer-readable storage medium according to claim 18 , wherein the calculating a difference feature of the marking patterns based on the marking patterns in adjacent images of the image sequence comprises:
determining two closest marking patterns from adjacent images at an i th frame and an (i+1) th frame in the image sequence as the same marking pattern, i being an integer; and
calculating the difference feature of the marking patterns based on at least one of locations and deformation of the marking patterns in the images at the i th frame and the (i+1) th frame.