Method for generating a bird's eye view image
A computer-implemented method for generating a bird's eye view image of a scene includes: (a) acquiring at least one lidar frame comprising points with inherent distance information and at least one camera image of the scene; (b) generating a mesh representation of the scene by using the at least one lidar frame, the mesh representation representing surfaces shown in the scene with inherent distance information; (c) generating a mask image by classifying pixels of the at least one camera image as representing ground pixels or non-ground pixels of the at least one camera image; and (d) generating the bird's eye view image by enhanced inverse perspective mapping exploiting distance information inherent to the surfaces of the mesh representation, pixels of the mask image classified as ground pixels, and the at least one camera image.
1. A computer-implemented method for generating a bird's eye view image of a scene, the method comprising the steps:
(a) acquiring, by a lidar sensor, at least one lidar frame comprising points with inherent distance information and acquiring, by a camera, at least one camera image of the scene;
(b) generating a mesh representation of the scene based on the at least one lidar frame, wherein the mesh representation represents surfaces shown in the scene with inherent distance information;
(c) generating a mask image, wherein generating the mask image comprises:
generating a segmented lidar frame, wherein generating the segmented lidar frame comprises classifying points of the at least one lidar frame as representing ground points or non-ground points of the scene based on the mesh representation; and
mapping the segmented lidar frame to the at least one camera image to classify pixels of the at least one camera image as representing ground pixels or non-ground pixels of the at least one camera image; and
(d) generating the bird's eye view image based on the mask image and the mesh representation, wherein generating the bird's eye view image comprises performing an inverse perspective mapping on the at least one camera image using the inherent distance information from the mesh representation, and wherein in the bird's eye view image non-ground objects are masked out based on the classification of pixels of the at least one camera image as representing ground pixels or non-ground pixels.
2. The method according to claim 1 , wherein in step (c), classifying the pixels of the at least one camera image are as representing ground pixels or non-ground pixels of the at least one camera image comprises using a k-nearest neighbors algorithm, wherein the use of the k-nearest neighbors algorithm takes into consideration an established correspondence between each pixel in the at least one camera image and each point in the segmented lidar frame.
3. The method according to claim 2 , wherein the k-nearest-neighbors algorithm uses k=3.
4. The method according to claim 1 , wherein in step (c) classifying the points using the mesh representation comprises the following steps:
calculating normal vectors and distances from points of the at least one lidar frame to the mesh representation; and
classifying a point in the at least one lidar frame as a ground point based on the calculated distance of the point being within a predetermined distance and the calculated normal vector of the point being upwardly directed.
5. The method according to claim 4 , wherein step (c) further comprises:
classifying remaining unclassified points of the at least one lidar frame, wherein classifying the remaining unclassified points of the at least one lidar frame comprises the following steps:
classifying a remaining point as a ground point based on a proportion of neighboring points classified as ground points exceeding a predetermined threshold, or based on a distance between the remaining point and other points classified as ground points being below a predetermined distance: or
classifying still remaining unclassified points as non-ground based on the nearest point classified as a ground point exceeding a predetermined distance or there being a point classified as a ground point located below it.
6. The method according to claim 1 , wherein in step (d) generating the bird's eye view image comprises the following steps:
calculating real-world coordinates of the surfaces as represented by the mesh representation;
projecting the real-world coordinates into an image space to generate pixels of the bird's eye view image, wherein projecting the real-world coordinates into the image space comprises transforming 3D coordinates into 2D pixels; and
coloring pixels of the bird's eye view image by referring to the classified pixels in the mask image and colors of the corresponding pixels in the at least one camera image.
7. The method according to claim 1 , further comprising the step:
(e) generating a trajectory bird's eye view image by merging a plurality of bird's eye view images which correspond to camera images taken by the same camera from different positions.
8. The method according to claim 7 , wherein merging the plurality of bird's eye view images comprises:
for a respective pixel of the trajectory bird's eye view image, selecting, from among corresponding pixels of the plurality of bird's eye view images, a pixel having an estimated shortest projection distance to the camera as a representative pixel for the trajectory bird's eye view image.
9. The method according to claim 7 , further comprising the step:
(f) generating a final bird's eye view image by merging a plurality of trajectory bird's eye view images, wherein respective trajectory bird's eye view images of the plurality of trajectory bird's eye view images correspond to camera images taken by different cameras.
10. The method according to claim 9 , wherein merging the plurality of trajectory bird's eye view images comprises equalizing mean and variance values of histograms of the trajectory bird's eye images.
11. The method according to claim 10 , wherein the histograms are color histograms.
12. The method according to claim 9 , wherein the plurality of bird's eye view images are merged such that the trajectory bird's eye view image is composed of those pixels of the plurality of bird's eye view images having an estimated shortest projection distance to the camera among corresponding pixels across the plurality of bird's eye view images; and
wherein merging the plurality of trajectory bird's eye view images comprises the following steps:
calculating respective projection errors corresponding to respective pixels of the plurality of trajectory bird's eye view images, wherein a respective projection error corresponding to a respective pixel represents a pixel value difference between corresponding pixels of different bird's eye view images corresponding to camera images taken by the same camera; and
composing the final bird's eye image using pixels of the plurality of trajectory bird's eye view images having the smallest projection error among corresponding pixels across the plurality of trajectory bird's eye view images.
13. A system for generating a bird's eye view image of a scene, the system comprising:
a lidar sensor configured to obtain at least one lidar frame comprising points with inherent distance information;
a camera configured to obtain at least one camera image of the scene; and
a processor configured to:
generate a mesh representation of the scene based on the at least one lidar frame, wherein the mesh representation represents surfaces shown in the scene with inherent distance information;
generate a mask image, wherein generating the mask image comprises:
generating a segmented lidar frame, wherein generating the segmented lidar frame comprises classifying points of the at least one lidar frame as representing ground points or non-ground points of the scene based on the mesh representation; and
mapping the segmented lidar frame to the at least one camera image to classify pixels of the at least one camera image as representing ground pixels or non-ground pixels of the at least one camera image by using the segmented lidar frame; and
generate the bird's eye view image based on the mask image and the mesh representation, wherein generating the bird's eye view image comprises performing an inverse perspective mapping on the at least one camera image using the inherent distance information from the mesh representation, and wherein in the bird's eye view image non-ground objects are masked out based on the classification of pixels of the at least one camera image as representing ground pixels or non-ground pixels.
14. The system according to claim 13 , wherein the processor is comprised in a test or simulation environment for testing autonomously driving vehicles or simulating traffic scenes and/or in a vehicle control system for autonomous driving and/or in a driver assistance system.
15. A non-transitory computer-readable medium having processor-executable instructions stored thereon for generating a bird's eye view image of a scene, wherein the processor-executable instructions, when executed, facilitate performance of the following steps:
(a) acquiring, by a lidar sensor, at least one lidar frame comprising points with inherent distance information and acquiring, by a camera, at least one camera image of the scene;
(b) generating a mesh representation of the scene based on the at least one lidar frame, wherein the mesh representation represents surfaces shown in the scene with inherent distance information;
(c) generating a mask image, wherein generating the mask image comprises:
generating a segmented lidar frame, wherein generating the segmented lidar frame comprises classifying points of the at least one lidar frame as representing ground points or non-ground points of the scene based on the mesh representation; and
mapping the segmented lidar frame to the at least one camera image to classify pixels of the at least one camera image as representing ground pixels or non-ground pixels of the at least one camera image; and
(d) generating the bird's eye view image based on the mask image and the mesh representation, wherein generating the bird's eye view image comprises performing an inverse perspective mapping on the at least one camera image using the inherent distance information from the mesh representation, and wherein in the bird's eye view image non-ground objects are masked out based on the classification of pixels of the at least one camera image as representing ground pixels or non-ground pixels.