Electronic device, system and method for determining a semantic grid of an environment of a vehicle
An electronic device for determining a semantic grid of an environment of a vehicle is provided. The electronic device is configured to: receive first image data of an optical sensor, the first image data comprising a 2D image of the environment, perform a semantic segmentation of the 2D image and project the resulting semantic image into at least one predetermined semantic plane, receive an occupancy grid representing an allocentric bird eye's view of the environment. The control device further includes a neural network configured to determine a semantic grid by fusing the occupancy grid with the at least one predetermined semantic plane.
1. An electronic device for determining a semantic grid of an environment of a vehicle, the electronic device being configured to:
receive first image data of an optical sensor, the first image data comprising a 2D image of the environment,
perform a semantic segmentation of the 2D image and project the resulting semantic image into at least one predetermined semantic plane to create a semantic plane,
receive an occupancy grid representing an allocentric bird eye's view of the environment, wherein
the control device further comprises:
a neural network configured to determine a semantic grid by fusing the occupancy grid with the semantic plane.
2. The electronic device according to claim 1 , wherein
the semantic image is projected to the semantic plane by transforming the coordinate system of the semantic image such that it matches with the coordinate system of the occupancy grid.
3. The electronic device according to claim 1 , wherein
the at least one predetermined semantic plane is parallel to an allocentric bird eye's view and/or parallel to the ground plane with a predetermined distance to the ground plane, and/or the at least one predetermined semantic plane overlaps the occupancy grid in the allocentric bird eye's view.
4. The electronic device according to claim 1 , wherein
the resulting semantic image is projected into a plurality of predetermined parallel semantic planes, each having a different predetermined distance to the ground plane.
5. The electronic device according to claim 1 , wherein
the 2D image is segmented by assigning each image pixel with a semantic label.
6. The electronic device according to claim 1 , wherein
the at least one semantic plane comprises a plurality of plane cells, each plane cell comprising a semantic label, and/or the semantic planes comprise each a plurality of plane cells, wherein each pixel of the semantic image is assigned to a plane cell in at least one of the semantic planes.
7. The electronic device according claim 1 , wherein
the occupancy grid comprises a plurality of occupancy grid cells, each indicating an occupancy state.
8. The electronic device according to claim 7 , wherein
the neural network is configured to:
fuse the occupancy grid with the plurality of predetermined semantic planes by estimating at least one of the plane cells in the semantic planes which matches with an occupancy grid cell.
9. The electronic device according to claim 8 , wherein
the neural network is configured to:
fuse the occupancy grid with the predetermined semantic planes by selecting a plane cell of the semantic planes based on the grid state of an overlapping occupancy grid cell, and
assigning an overlapping occupancy grid cell with the semantic label of the selected plane cell.
10. The electronic device according to claim 1 , wherein
the control device further comprises:
a further neural network configured to perform a semantic segmentation of the 2D image.
11. The electronic device according to claim 1 , wherein
the neural network is a convolutional-deconvolutional geometry fusion network.
12. The electronic device according to claim 1 , wherein
the semantic grid comprises a plurality of cells, each cell being assigned with a semantic information, the semantic information in particular comprises a semantic class and a probability for each class.
13. A system for determining a semantic grid of an environment of a vehicle, comprising:
an electronic device according to claim 1 ,
an optical sensor, configured to generate first image data comprising a 2D image of the environment, and
an active sensor configured to generate second data comprising a 3D point cloud of the environment representing a plurality of scan areas distributed over the environment.
14. The system according to claim 13 , wherein
the optical sensor is a monocular camera or a set of monocular cameras.
15. A method of determining a semantic grid of an environment of a vehicle, comprising the steps of:
receiving first image data of an optical sensor, the first image data comprising a 2D image of the environment,
performing a semantic segmentation of the 2D image and projecting the resulting semantic image into at least one predetermined semantic plane to create a semantic plane,
receiving an occupancy grid representing an allocentric bird eye's view of the environment, wherein
a neural network determines a semantic grid by fusing the occupancy grid with the semantic plane.