Urban environment labelling
The present invention relates to a method and system for automatic localisation of static objects in an urban environment. More particularly, the present invention relates to the use of noisy 2-Dimensional (2D) image data to identify and determine 3-Dimensional (3D) positions of objects in large scale urban or city environments. Aspects and/or embodiments seek to provide a method, system, and vehicle for automatically locating static 3D objects in urban environments by using a voting-based triangulation technique. Aspects and/or embodiments also provide a method for updating map data after automatically new 3D static objects in an environment.
1. A method for locating one or more static objects in an environment, the method comprising;
receiving a plurality of 2D images of the environment;
detecting the one or more static objects from the plurality of 2D images;
generating a data set of 2D static object detections associated with the one or more static objects; and
performing a voting-based triangulation on the data set of 2D static object detections to determine one or more 3D positions for the one or more static objects detected in the environment, wherein a vote to determine the one or more 3D positions is based on a distance between an estimated 3D position for one of the one or more static objects and a position of one of the 2D static object detections satisfying a distance threshold.
2. The method of claim 1 wherein the plurality of 2D images care associated with at least one of or any combination of: camera-intrinsic parameters;
camera pose information; six degrees-of-freedom pose information for each image; or being captured using a fleet of vehicles.
3. The method of claim 1 wherein the detecting the one or more static objects comprises at least one of or any combination of:
detecting one of the one or more static objects in at least two 2D images from the plurality of 2D images;
detecting the one of the one or more static objects in the at least two 2D images with a minimum angle difference;
computing a pixel probability to determine whether a pixel corresponds to the one of the one or more static objects;
determining neighbouring connecting pixels that represent the one of the one or more static objects; or
identifying the one or more static objects in each 2D image.
4. The method of claim 1 wherein the performing the voting-based triangulation comprises using at least two detections from different 2D images that correspond to the one of the one or more static objects.
5. The method of claim 4 wherein the performing the voting-based triangulation further comprises creating a hypothesis for a pair of static object detections.
6. The method of claim 5 wherein the hypothesis is scored based on a number of votes wherein the votes are indicative of a correctness of the hypothesis.
7. The method of claim 5 wherein the estimated 3D position is estimated based on the hypothesis.
8. The method of claim 7 further comprising performing a K-view triangulation to determine an accurate 3D position for the hypothesis, wherein K is dependent upon a number of different images that contributed a vote to the hypothesis.
9. The method of claim 5 wherein the performing the voting-based triangulation comprises:
using image pose information and camera-intrinsic information to project the estimated 3D position into the plurality of 2D images.
10. The method of claim 5 wherein the hypothesis is confirmed to be correct by a number of received votes, and the hypothesis indicates a correct 3D position of the one of the one or more static objects.
11. The method of claim 5 wherein the creating the hypothesis comprises any one of:
determining whether one or more projected positions are less than d max , to any 2D detection;
determining whether a 3D position point is triangulated in front of a camera;
determining whether an angle between the one or more projected positions are larger than θ min ; or
determining whether a distance from the one or more static objects to the camera is less than r max .
12. The method of claim 5 wherein the vote indicates the hypothesis is correct when the distance between the estimated 3D position for the one of the one or more static objects and the position of the one of the 2D static object detections satisfies the distance threshold and the vote indicates the hypothesis is incorrect when the distance between the estimated 3D position for the one of the one or more static objects and the position of the one of the 2D static object detections fails to satisfy the distance threshold.
13. The method of claim 1 wherein the one or more static objects are traffic lights and/or traffic signs.
14. The method of claim 1 wherein the method for locating one or more static objects in an environment is performed in clusters of the environment using a distribution schema to split the data set.
15. The method of claim 1 wherein the detecting one or more static objects is based on a binary segmentation network and/or convolutional neural networks.
16. The method of claim 15 wherein the binary segmentation network and/or the convolutional neural networks are trained using an existing data set.
17. The method of claim 1 wherein each 2D image is processed using structure-from-motion, SFM, techniques to estimate pose information of each 2D image.
18. A system for locating one or more static objects in an environment, the system comprising a processor that causes the system to perform:
receive a plurality of 2D images of the environment;
detect the one or more static objects from the plurality of 2D images;
generate a data set of 2D static object detections associated with the one or more static objects; and
perform a voting-based triangulation on the data set of the 2D static object detections to determine one or more 3D positions for the one or more static objects detected in the environment, wherein a vote to determine the one or more 3D positions is based on a distance between an estimated 3D position for one of the one or more static objects and a position of one of the 2D static object detections satisfying a distance threshold.
19. The system of claim 18 , wherein the processor causes the system to further perform process a portion of the environment based on a plurality of independent clusters.
20. A non-transitory computer-readable medium comprising instructions which, when executed by a computer, cause the computer to perform a method comprising:
receiving a plurality of 2D images of the environment;
detecting one or more static objects from the plurality of 2D images;
generating a data set of 2D static object detections associated with the one or more static objects; and
performing a voting-based triangulation on the data set of 2D static object detections to determine one or more 3D positions for the one or more static objects detected in the environment, wherein a vote to determine the one or more 3D positions is based on a distance between an estimated 3D position for one of the one or more static objects and a position of one of the 2D static object detections satisfying a distance threshold.