Granted Patent B2

US 12,228,409 · App. 18/539,544 · Granted Feb 18, 2025

Inventors: Yi Luo (San Diego, CA); Yi Wang (San Diego, CA); Ke Xu (San Diego, CA)

Assignee: TUSIMPLE, INC.

G01C21/1656G01C21/1652G01S17/86G01S17/89G01S17/931G06F18/24G06F18/2415G06V10/757G06V10/764G06V20/56G06T17/05

Abstract

Claims (40)

1. A method of localization comprising

processing of images from a camera and processing a data from a light detection and ranging (LiDAR), the processing comprising:

constructing a 3D submap using images from the camera and vehicle pose information;

constructing a 3D global map generated from the data from the LiDAR and vehicle pose information;

computing the location of the 3D submap in the global map, wherein the location of the 3D submap includes center position of the 3D submap; and

aligning the 3D submap with the global map.

2. The method according to claim 1 , wherein the vehicle pose information comprises vehicle position and orientation information.

3. The method according to claim 1 , wherein the 3D submap is constructed by means of visual simultaneous localization and mapping (SLAM).

4. The method according to claim 1 , wherein constructing the 3D global map comprises constructing a city-scale 3D map based on the data from the LiDAR using LiDAR mapping.

5. The method according to claim 1 , wherein the data from the LiDAR was gathered by a LiDAR with a plurality of simultaneous rotating beams at varying angles.

6. The method according to claim 1 , wherein at least one coordinate of the 3D submap is transformed to at least one coordinate of the global map, to obtain a coarse location of the 3D submap in the global map.

7. A method of localization comprising

processing of images from a camera and processing a data from a light detection and ranging (LiDAR), the processing comprising:

performing alignment of the data, which includes sensor calibration and time synchronization;

extracting first features from a 3D submap, wherein the 3D submap is generated using the images from the camera;

extracting second features from a global map, wherein the global map is generated from the data from the LiDAR;

generating matching scores from comparing the first features to the second features, wherein each matching score represents a correspondence between one of the first features and one of the second features, and wherein each matching score includes a distance between the one of the first features and the one of the second features; and

removing one or more correspondences between the first features and the second features for one or more matching scores that are larger than a threshold value.

8. The method according to claim 7 , wherein before generating matching scores, the processing of the images further comprises:

transforming at least one coordinate of the 3D submap to at least one coordinate of the global map to obtain a coarse location of the 3D submap in the global map; and

utilizing the coarse location of the 3D submap within the global map to facilitate refinement of feature correspondence.

9. The method according to claim 7 , wherein the distance between the one of the first features and the one of the second features is determined by a trained classifier.

10. The method according to claim 7 , wherein the extracting the first features from the 3D submap and the extracting the second features from the global map comprises:

extracting structured features and unstructured features from the 3D submap and the global map.

11. The method according to claim 7 , wherein the data from the LiDAR was gathered by a LiDAR with a plurality of simultaneous rotating beams at varying angles.

12. The method according to claim 8 , wherein the distance between the one of the first features and the one of the second features is determined by a trained classifier.

13. A method of localization comprising

processing of images from a camera and processing a data from a light detection and ranging (LiDAR), the processing comprising:

constructing a 3D submap using images from the camera;

constructing a global map generated from the data from the LiDAR;

obtaining center position of the 3D submap;

transforming at least one coordinate of the 3D submap into at least one coordinate of the global map;

aligning the 3D submap with the global map; and

extracting features from the 3D submap and the global map.

14. The method according to claim 13 , wherein after extracting the features from the 3D submap and the global map, the processing of the images further comprises:

classifying the extracted features into classes; and

establishing correspondence of features in a same class between the 3D submap and the global map.

15. The method according to claim 13 , wherein the features extracted from the 3D submap and the global map comprise structured features and unstructured features from the 3D submap and the global map.

16. The method according to claim 14 , wherein after establishing correspondence of features in a same class between the 3D submap and the global map, feature correspondences are removed based on the alignment of the 3D submap with the global map.

17. The method according to claim 16 , wherein after feature correspondences are removed based on the alignment of the 3D submap with the global map, an iterative estimation of a location of the 3D submap is performed based on the center position of the 3D submap.

Assignments (3)

Continuity (4)

References Cited (329)