Granted Patent B2

US 12,243,321 · App. 17/457,407 · Granted Mar 4, 2025

Inventor: Jakub Derbisz (Cracow, PL)

Assignee: APTIVE TECHNOLOGIES AG

G06V20/56G06F18/2431G06T7/70G06V10/95G06T2207/30252

Abstract

Claims (85)

1. A method, comprising:

capturing, via a camera of a vehicle in an environment, a two dimensional visual image of the environment;

determining, with a processor of the vehicle, within the visual image, a limitation of a free space with respect to the vehicle;

capturing, via a sensor of the vehicle, for objects in the environment, three dimensional distance data with respect to the vehicle;

based on assigning the distance data to the visual image, transferring the limitation of the free space from the visual image to a bird's-eye view, with respect to the vehicle;

determining, for each object identified in the visual image, a bounding box and a classification;

selecting objects identified in the visual image that are limiting the free space;

assigning the bounding box of each of the objects selected to the limitation of the free space transferred to the bird's-eye view; and

according to the classification of each of the objects selected, classifying segments of the limitation of the free space transferred to the bird's-eye view, thereby determining a semantic free space in the environment of the vehicle,

wherein determining the limitation of the free space comprises determining the limitation of the free space based on executing a border following algorithm to determine the limitation of the free space as being a continuously extending contour of the free space.

2. The method according to claim 1 ,

wherein determining the limitation of the free space comprises determining the limitation of the free space based on applying a first neural network to the visual image.

3. The method according to claim 1 ,

wherein assigning the distance data to the visual image comprises applying a projecting transformation of the three dimensional distance data to the two dimensional visual image.

4. The method according to claim 3 ,

wherein transferring the limitation of the free space from the visual image to the bird's-eye view comprises applying an inverse transformation of the projecting transformation to the limitation of the free space.

5. The method according to claim 1 ,

wherein the limitation of the free space comprises a plurality of limitation points represented in the visual image, and transferring the limitation of the free space from the visual image to the bird's-eye view comprises:

selecting a predetermined number of closest points from distance data for each of the limitation points; and

estimating a distance, with respect to the vehicle, for each of the limitation points as an average of distances, with respect to the vehicle, for each of the closest points.

6. The method according to claim 1 ,

wherein determining the bounding box and the classification for each of the objects selected comprises applying a second neural network to the visual image.

7. The method according to claim 6 ,

wherein determining the classification comprises estimating a class probability for the bounding box for each of the objects selected when applying the second neural network.

8. The method according to claim 7 , further comprising:

determining a certainty score for each segment of the limitation of the free space based on the class probability for the bounding box for each of the objects selected.

9. The method according to claim 1 , further comprising:

determining, with respect to the vehicle, based on the three dimensional distance data, a distance of a center for the bounding box for each of the objects selected; and

assigning the distance of the center for the bounding box for each of the objects selected to the limitation of the free space in the bird's-eye view.

10. The method according to claim 9 ,

wherein determining the distance of the center for the bounding box for each of the objects selected comprises determining the distance of the center for the bounding box for each of the objects selected based on a predetermined number of assigned distance data being closest to the center of that bounding box.

11. The method according to claim 10 ,

wherein assigning the distance of the center for the bounding box for each of the objects selected to the limitation of the free space in the bird's-eye view comprises assigning the distance of the center for the bounding box for each of the objects selected only if the distance of the center of the bounding box to at least one of the assigned distance data being less than a predetermined distance.

12. The method according to claim 1 , further comprising:

dividing the segments of the limitation of the free space equally by a fixed and predetermined azimuth angle with respect to the vehicle.

13. A system, comprising:

a processor for a vehicle, the processor configured to:

capture, via a camera for the vehicle, a two dimensional visual image of an environment;

determine, within the visual image, a limitation of a free space with respect to the vehicle;

capture, via a sensor for the vehicle, for objects in the environment, three dimensional distance data with respect to the vehicle;

based on assigning the distance data to the visual image, transfer the limitation of the free space from the visual image to a bird's-eye view, with respect to the vehicle;

determine, for each object identified in the visual image, a bounding box and a classification;

select objects identified in the visual image that are limiting the free space;

assign the bounding box of each of the objects selected to the limitation of the free space transferred to the bird's-eye view; and

classify segments of the limitation of the free space transferred to the bird's-eye view according to the classification of each of the objects selected to determine a semantic free space in the environment of the vehicle,

wherein determining the limitation of the free space comprises determining the limitation of the free space based on executing a border following algorithm to determine the limitation of the free space as being a continuously extending contour of the free space.

14. The system of claim 13 , further comprising:

the camera for the vehicle; and

the sensor for the vehicle.

15. The system of claim 14 , further comprising the vehicle, wherein the processor is configured as a free space module to output the semantic free space determined in the environment of the vehicle.

16. A non-transitory computer readable storage medium comprising instructions, that when executed, configure a processor for a vehicle to:

capture, via a camera for the vehicle, a two dimensional visual image of an environment;

determine, within the visual image, a limitation of a free space with respect to the vehicle;

capture, via a sensor for the vehicle, for objects in the environment, three dimensional distance data with respect to the vehicle;

based on assigning the distance data to the visual image, transfer the limitation of the free space from the visual image to a bird's-eye view, with respect to the vehicle;

determine, for each object identified in the visual image, a bounding box and a classification;

select objects identified in the visual image that are limiting the free space;

assign the bounding box of each of the objects selected to the limitation of the free space transferred to the bird's-eye view; and

classify segments of the limitation of the free space transferred to the bird's-eye view according to the classification of each of the objects selected to determine a semantic free space in the environment of the vehicle,

wherein determining the limitation of the free space comprises determining the limitation of the free space based on executing a border following algorithm to determine the limitation of the free space as being a continuously extending contour of the free space.

17. The computer readable storage medium according to claim 16 ,

wherein the instructions, when executed, configure the processor to determine the limitation of the free space by determining the limitation of the free space based on applying a first neural network to the visual image.

18. The computer readable storage medium according to claim 16 ,

wherein the instructions, when executed, configure the processor to assign the distance data to the limitation of the free space by applying a projecting transformation of the three dimensional distance data to the two dimensional visual image.

19. A method, comprising:

capturing, via a camera of a vehicle in an environment, a two dimensional visual image of the environment;

determining, with a processor of the vehicle, within the visual image, a limitation of a free space with respect to the vehicle;

capturing, via a sensor of the vehicle, for objects in the environment, three dimensional distance data with respect to the vehicle;

based on assigning the distance data to the visual image, transferring the limitation of the free space from the visual image to a bird's-eye view, with respect to the vehicle;

determining, for each object identified in the visual image, a bounding box and a classification;

selecting objects identified in the visual image that are limiting the free space;

assigning the bounding box of each of the objects selected to the limitation of the free space transferred to the bird's-eye view;

according to the classification of each of the objects selected, classifying segments of the limitation of the free space transferred to the bird's-eye view, thereby determining a semantic free space in the environment of the vehicle;

determining, with respect to the vehicle, based on the three dimensional distance data, a distance of a center for the bounding box for each of the objects selected; and

assigning the distance of the center for the bounding box for each of the objects selected to the limitation of the free space in the bird's-eye view.

20. A method, comprising:

capturing, via a camera of a vehicle in an environment, a two dimensional visual image of the environment;

determining, with a processor of the vehicle, within the visual image, a limitation of a free space with respect to the vehicle;

capturing, via a sensor of the vehicle, for objects in the environment, three dimensional distance data with respect to the vehicle;

based on assigning the distance data to the visual image, transferring the limitation of the free space from the visual image to a bird's-eye view, with respect to the vehicle;

determining, for each object identified in the visual image, a bounding box and a classification;

selecting objects identified in the visual image that are limiting the free space;

assigning the bounding box of each of the objects selected to the limitation of the free space transferred to the bird's-eye view;

according to the classification of each of the objects selected, classifying segments of the limitation of the free space transferred to the bird's-eye view, thereby determining a semantic free space in the environment of the vehicle; and

dividing the segments of the limitation of the free space equally by a fixed and predetermined azimuth angle with respect to the vehicle.

Assignments (4)

Priority Claims (1)

Continuity (1)

References Cited (65)