Method and apparatus with lane generation
A method of generating lane information using a neural network includes generating a lane probability map based on an input image, generating lane feature information and depth feature information by applying the lane probability map to a second neural network, generating depth distribution information by applying the depth feature information to a third neural network, generating spatial information based on the lane feature information and the depth distribution information, generating offset information including a displacement between a position of a lane and a reference line by applying the spatial information to a fourth neural network, and generating three-dimensional (3D) lane information using the offset information.
1. A processor-implemented method of generating lane information using a neural network, comprising:
generating a lane probability map based on an input image;
generating lane feature information and depth feature information by applying the lane probability map to a second neural network;
generating depth distribution information by applying the depth feature information to a third neural network;
generating spatial information based on the lane feature information and the depth distribution information;
generating offset information comprising a displacement between a position of a lane and a reference line by applying the spatial information to a fourth neural network; and
generating three-dimensional (3D) lane information using the offset information.
2. The method of claim 1 , wherein the generating of the lane probability map comprises:
generating the lane probability map using a first neural network trained to generate lane probability values respectively corresponding to pixels in the input image, wherein each of the lane probability values comprise a probability of a pixel of the pixels being in the lane.
3. The method of claim 1 , wherein the second neural network is trained to generate the lane feature information using a first channel associated with lane features, or to generate the depth feature information using a second channel associated with depth features.
4. The method of claim 1 , wherein the depth feature information comprises at least one of a height of the lane comprising a length in a vertical direction with respect to the reference line or a distance between an imaging device and the lane.
5. The method of claim 1 , wherein the depth distribution information comprises depth probability distributions respectively corresponding to pixels in the input image.
6. The method of claim 1 , wherein the generating of the spatial information comprises:
generating frustum-based first spatial information through an outer product of the lane feature information and the depth distribution information.
7. The method of claim 1 , wherein the generating of the spatial information comprises:
generating frustum-based first spatial information based on the lane feature information and the depth distribution information;
transforming the first spatial information into voxel map-based second spatial information;
transforming the second spatial information into third spatial information based on channel dimension information; and
determining the third spatial information to be the spatial information.
8. The method of claim 7 , wherein the first spatial information comprises spatial information in a 3D coordinate system based on a viewpoint of the input image.
9. The method of claim 7 , wherein the second spatial information comprises spatial information in a 3D coordinate system based on a viewpoint of a vehicle.
10. The method of claim 7 , wherein the transforming into the second spatial information comprises:
transforming the first spatial information into the second spatial information based on calibration parameters,
wherein the calibration parameters comprise any one or any combination of intrinsic parameters of an imaging device for capturing the input image and extrinsic parameters of the imaging device.
11. The method of claim 10 , wherein the intrinsic parameters comprise one or any combination of focal length, a principal point, an aspect ratio, a center point, and an asymmetry parameter of the imaging device and the extrinsic parameters comprise an installation height of the imaging device, a pan of the imaging device, and a tilt of the imaging device.
12. The method of claim 1 , wherein the offset information comprises at least one of a first offset in a direction horizontal to a travel direction of a vehicle or a second offset in a direction vertical to the reference line.
13. The method of claim 1 , further comprising:
generating at least one of a probability value of the reference line being the lane, visibility information, a first uncertainty corresponding to a first offset, and a second uncertainty corresponding to a second offset, by applying the spatial information to the fourth neural network.
14. The method of claim 1 , wherein the neural network is trained using a loss function defined based on any one or any combination of a first uncertainty and a second uncertainty.
15. The method of claim 1 , wherein the neural network is trained using a loss function defined based on any one or any combination of a first uncertainty associated with an offset in a horizontal direction and a second uncertainty associated with an offset in a vertical direction.
16. The method of claim 1 , wherein the neural network comprises a neural network trained using a loss function defined based on a probability value of the reference line being the lane.
17. The method of claim 1 , wherein the neural network comprises a neural network trained using a loss function defined based on visibility information.
18. The method of claim 1 , wherein the first neural network comprises a neural network trained using images labelled with lanes as ground truth data.
19. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 .
20. A device comprising:
a processor configured to
generate a lane probability map based on an input image,
generate lane feature information and depth feature information by applying the lane probability map to a second neural network,
generate depth distribution information by applying the depth feature information to a third neural network,
generate spatial information based on the lane feature information and the depth distribution information,
generate offset information comprising a displacement between a position of a lane and a reference line by applying the spatial information to a fourth neural network, and
generate three-dimensional (3D) lane information using the offset information.
21. A vehicle comprising:
a sensor configured to capture an input image;
a processor configured to execute instructions stored in a non-transitory computer-readable storage medium to
generate a lane probability map based on the first neural network trained to generate lane probability values respectively corresponding to pixels in the input image,
generate lane feature information and depth feature information by applying the lane probability map to the second neural network,
generate depth distribution information by applying the depth feature information to the third neural network,
generate spatial information based on the lane feature information and the depth distribution information, and
generate offset information comprising a displacement between a position of a lane and a reference line by applying the spatial information to the fourth neural network,
generate three-dimensional (3D) lane information using the offset information, and
output the 3D lane information to a vehicle control unit (VCU) of the vehicle.
22. The method of claim 21 , wherein the second neural network comprises a first sub-neural network trained to generate the lane feature information, and a second sub-neural network trained to generate the depth features, and the first sub-neural network and the second sub-neural network share one or more layers.