Systems and methods for visuo-tactile object pose estimation
Systems and methods for visuo-tactile object pose estimation are provided. In one embodiment, a method includes receiving image data about an object and receiving depth data about the object. The method also includes generating a visual estimate of the object based on the image data and the depth data. The method further includes receiving tactile data about the object. The method yet further includes generating a tactile estimate of the object based on the tactile data. The method includes estimating a pose of the object based on the visual estimate and the tactile estimate.
1. A system for visuo-tactile object pose estimation, comprising:
a sensor module, implemented via a processor, configured to:
receive image data about an object in an environment;
receive depth data about the object in the environment; and
receive tactile data about the object in the environment;
a visual module, implemented via the processor, configured to generate a visual estimate of the object by processing a combined dataset of the image data and the depth data into a reconstructed fused image;
a tactile module, implemented via the processor, configured to separately generate a tactile estimate of the object by processing the tactile data; and
a pose module, implemented via the processor, configured to estimate a pose of the object based on the visual estimate and the tactile estimate, wherein
the visual module employs a first convolutional neural network (CNN) to process the combined dataset of the image data and the depth data.
2. The system of claim 1 , wherein the image data is RGB data from a first optical sensor.
3. The system of claim 1 , wherein the depth data is received from a second sensor that is a ranging sensor.
4. The system of claim 1 , wherein the tactile module employs a second CNN, and wherein the pose module employs a fully connected CNN layer.
5. The system of claim 1 , wherein the visual module, implemented via the processor, is further configured to determine a region of interest (RoI) based on the image data.
6. The system of claim 5 , wherein the RoI is determined based on an object segmentation neural network.
7. A method for visuo-tactile object pose estimation, comprising:
receiving image data about an object in an environment;
receiving depth data about the object;
generating a visual estimate of the object by processing a combined dataset of the image data and the depth data into a reconstructed fused image;
receiving tactile data about the object;
generating, separately, a tactile estimate of the object by processing the tactile data; and
estimating a pose of the object based on the visual estimate and the tactile estimate, wherein
the visual estimate is generated based on a first convolutional neural network (CNN) to process the combined dataset of the image data and the depth data.
8. The method of claim 7 , wherein the image data is RGB data from a first optical sensor.
9. The method of claim 7 , wherein the depth data is received from a second sensor that is a ranging sensor.
10. The method of claim 7 , wherein the tactile estimate is generated based on a second CNN, and wherein the pose is estimated based on a fully connected CNN layer.
11. The method of claim 7 , the method further comprising determining a region of interest (RoI) based on the image data.
12. The method of claim 11 , wherein the RoI is determined based on an object segmentation neural network.
13. The method of claim 7 , wherein the pose of the object defines a location of the object in a three-dimensional space of the environment.
14. A non-transitory computer readable storage medium storing instructions that when executed by a computer having a processor to perform a method for visuo-tactile object pose estimation, the method comprising:
receiving image data about an object in an environment;
receiving depth data about the object;
generating a visual estimate of the object by processing a combined dataset of the image data and the depth data into a reconstructed fused image;
receiving tactile data about the object;
generating, separately, a tactile estimate of the object by processing the tactile data; and
estimating a pose of the object based on the visual estimate and the tactile estimate, wherein
the visual estimate is generated based on a first convolutional neural network (CNN) to process the combined dataset of the image data and the depth data.
15. The non-transitory computer readable storage medium of claim 14 , wherein the image data is RGB data from a first optical sensor.
16. The non-transitory computer readable storage medium of claim 14 , wherein the depth data is received from a second sensor that is a ranging sensor.
17. The non-transitory computer readable storage medium of claim 14 , wherein the tactile estimate is generated based on a second CNN, and wherein the pose is estimated based on a fully connected CNN layer.
18. The non-transitory computer readable storage medium of claim 14 , the method further comprising determining a region of interest (RoI) based on the image data.
19. The non-transitory computer readable storage medium of claim 18 , wherein the RoI is determined based on an object segmentation neural network.
20. The non-transitory computer readable storage medium of claim 14 , wherein the pose of the object defines a location of the object in a three-dimensional space of the environment.