Multi-modal data fusion for enhanced 3D perception for platforms

1. A method for providing a real time, three-dimensional (3D) navigational map for platforms, the method comprising integrating at least two sources of multi-modal and multi-dimensional platform sensor information to produce a more accurate 3D navigational map by:

receiving both a 3D point cloud from a first sensor on a platform with a first modality and a 2D image from a second sensor on the platform with a second modality different from the first modality;

generating at least one first space semantic label and at least one first space semantic label uncertainty associated with at least one first space point in the 3D point cloud;

generating at least one second space semantic label and at least one second space semantic label uncertainty associated with at least one second space point in the 2D image; and

determining if the at least one first space point and the at least one second space point represent a common point in a space surrounding the platform and, if so, fusing the at least one first space semantic label and the at least one first space semantic label uncertainty with the at least one second space semantic label and the at least one second space semantic label uncertainty to create fused 3D spatial information of the common point in the space surrounding the platform to enhance the 3D navigational map.

2. The method of claim 1 , further comprising:

fusing temporal or motion related information from the first sensor or the second sensor into the fused 3D spatial information.

3. The method of claim 1 , further comprising:

performing a subsequent fusion of geo-referenced data, distance information, or velocity information with the fused 3D spatial information to further enhance the 3D navigational map.

4. The method of claim 3 , further comprising:

periodically performing a joint inference on new and existing information from the first sensor and the second sensor to update the 3D navigational map.

5. The method of claim 1 , further comprising:

determining depth information from information from the first sensor or from information from the second sensor.

6. The method of claim 1 , further comprising:

fusing 3D information from the first sensor and 2D information from the second sensor using a joint deep-learning based 3D semantic segmentation process.

7. The method of claim 1 , further comprising:

using combined information from the first sensor and information from the second sensor to reason a complete and dense 3D scene by depth propagation or occlusion fill-in processes to enhance the 3D navigational map.

8. The method of claim 1 , further comprising:

localizing the 3D navigational map using a GPS source or an IMU source.

9. The method of claim 1 , further comprising:

using velocity information from a full motion video (FMV) LIDAR sensor to augment a motion estimation of the platform.

10. The method of claim 1 , further comprising:

periodically performing semantic segmentation over new and existing 3D processed information from the first sensor or information from the second sensor to update the 3D navigational map.

11. A method for determining semantic segmentation information of a common multi-dimensional space, the method comprising combining semantic segmentation information from a first multi-dimensional space with semantic segmentation information from a second multi-dimensional space to enhance the semantic segmentation information of the common multi-dimensional space by:

semantically segmenting at least one first multi-dimensional space point from a first multi-dimensional space to determine at least one first semantic label and at least one first semantic label uncertainty for the at least one first multi-dimensional space point;

semantically segmenting at least one second multi-dimensional space point from a second multi-dimensional space with a different multi-dimension than the first multi-dimensional space to determine at least one second semantic label and at least one second semantic label uncertainty for the at least one second multi-dimensional space point; and

determining if the at least one first multi-dimensional space point and the at least one second multi-dimensional space point represent a common multi-dimensional space point in the common multi-dimensional space and, if so, fusing the at least one first semantic label and the at least one first semantic label uncertainty with the at least one second semantic label and the at least one second semantic label uncertainty to create fused multi-dimensional spatial information to enhance a semantic segmentation of the common multi-dimensional space point in the common multi-dimensional space.

12. The method of claim 11 , wherein the first multi-dimensional space is a two-dimensional space, the second multi-dimensional space is a three-dimensional space, and the common multi-dimensional space is a three-dimensional space.

13. The method of claim 12 , wherein the at least one first multi-dimensional space point is extracted from a two-dimensional image and the at least one second multi-dimensional space point is extracted from a three-dimensional point cloud.

14. A non-transitory computer-readable medium having stored thereon at least one program, the at least one program including instructions which, when executed by a processor, cause the processor to perform a method for providing a real time, three-dimensional (3D) navigational map for platforms, the method comprising integrating at least two sources of multi-modal and multi-dimensional platform sensor information to produce a more accurate 3D navigational map by:

receiving both a 3D point cloud from a first sensor on a platform with a first modality and a 2D image from a second sensor on the platform with a second modality different from the first modality;

generating at least one first space semantic label and at least one first space semantic label uncertainty associated with at least one first space point in the 3D point cloud;

generating at least one second space semantic label and at least one second space semantic label uncertainty associated with at least one second space point in the 2D image; and

determining if the at least one first space point and the at least one second space point represent a common point in a space surrounding the platform and, if so, fusing the at least one first space semantic label and the at least one first space semantic label uncertainty with the at least one second space semantic label and the at least one second space semantic label uncertainty to create fused 3D spatial information of the common point in the space surrounding the platform to enhance the 3D navigational map.

15. The non-transitory computer-readable medium of claim 14 , further comprising:

incorporating temporal or motion related information from the first sensor or the second sensor into the fused 3D spatial information.

16. The non-transitory computer-readable medium of claim 14 , further comprising:

performing a subsequent fusion of geo-referenced data, distance information, or velocity information with the fused 3D spatial information.

17. The non-transitory computer-readable medium of claim 14 , further comprising:

periodically performing a joint inference on new and existing information from the first sensor and the second sensor to update the 3D navigational map.

18. The non-transitory computer-readable medium of claim 14 , further comprising:

determining depth information from information from the first sensor or from information from the second sensor.

19. The non-transitory computer-readable medium of claim 14 , further comprising:

fusing 3D information from the first sensor and 2D information from the second sensor using a joint deep-learning based 3D semantic segmentation process.

20. The non-transitory computer-readable medium of claim 14 , further comprising:

using combined information from the first sensor and information from the second sensor to reason a complete and dense 3D scene by depth propagation or occlusion fill-in processes to enhance the 3D navigational map.