Granted Patent B2

US 12,347,209 · App. 17/884,716 · Granted Jul 1, 2025

Inventors: Mina Alibeiginabi (Gothenburg, SE); Erik Brorsson (Gothenburg, SE); Silas Ulander (Lerum, SE); Benjamin Waubert (Gothenburg, SE)

Assignee: ZENSEACT AB

G06V20/588G06V10/7753G06V10/82G06V20/64

Abstract

Claims (35)

1. A method for training an artificial neural network configured for 3D lane detection based on unlabelled image data from a vehicle-mounted camera, the method comprising:

generating, by means of the artificial neural network, a first set of 3D lane boundaries in a first coordinate system based a first image captured by the vehicle-mounted camera;

generating, by means of the artificial neural network, a second set of 3D lane boundaries in a second coordinate system based on a second image captured by the vehicle mounted camera, wherein the second image is captured at a later moment in time compared to the first image and wherein the first image and the second image contain at least partly overlapping road portions;

transforming at least one of the second set of 3D lane boundaries and the first set of 3D lane boundaries based on positional data associated with the first image and the second image, such that the first set of 3D lane boundaries and the second set of 3D lane boundaries have a common coordinate system;

evaluating the first set of 3D lane boundaries against the second set of 3D lane boundaries in the common coordinate system in order to find matching lane pairs of the first set of 3D lane boundaries and the second set of 3D lane boundaries; and

updating one or more model parameters of the artificial neural network based on a spatio-temporal consistency loss between the found matching lane pairs of the first set of 3D lane boundaries and the second set of 3D lane boundaries.

2. The method according to claim 1 , wherein the step of updating one or more parameters of the artificial neural network comprises penalizing the artificial neural network based on the spatio-temporal consistency loss between matching lane pairs of the first set of 3D lane boundaries and the second set of 3D lane boundaries.

3. The method according to claim 1 , wherein the spatio-temporal consistency loss is based on a Euclidean distance between matching lane pairs of the first set of 3D lane boundaries and the second set of 3D lane boundaries.

4. The method according to claim 1 , wherein the step of evaluating the first set of 3D lane boundaries against the second set of 3D lane boundaries in the common coordinate system comprises:

determining an average distance between each lane of the first set of 3D lane boundaries and each lane of the second set of 3D lane boundaries in the common coordinate system;

comparing the determined average distances against an average distance threshold; and

determining each lane pair having a determined average distance below the average distance threshold as a matching lane pair.

5. The method according to claim 1 , wherein the first image and the second image are captured at different geographical positions.

6. The method according to claim 1 , wherein the artificial neural network is configured to generate a set of anchor lines where each anchor line is associated with a confidence value indicative of a presence of a lane boundary at the corresponding anchor line.

7. The method according to claim 6 , further comprising:

disregarding any anchor line associated with a confidence value below a confidence value threshold in the transformation step.

8. The method according to claim 6 , wherein the step of updating the one or more model parameters of the artificial neural network is further based on the confidence values of any anchor lines associated with matching lane pairs and any anchor lines associated with non-matching lane pairs, such that any anchor lines associated with matching lane pairs should predict a normalized confidence level of 1,0 and any anchor lines associated with non-matching lane pairs should predict a normalized confidence value of 0.

9. A non-transitory computer-readable storage medium storing one or more instructions configured to be executed by one or more processors of a processing system, the one or more instructions for performing the method according to claim 1 .

10. An apparatus for training an artificial neural network configured for 3D lane detection based on unlabelled image data from a vehicle-mounted camera, the apparatus comprising control circuitry configured to:

generate, by means of the artificial neural network, a first set of 3D lane boundaries in a first coordinate system based a first image captured by the vehicle-mounted camera;

generate, by means of the artificial neural network, a second set of 3D lane boundaries in a second coordinate system based on a second image captured by the vehicle mounted camera, wherein the second image is captured at a later moment in time compared to the first image and wherein the first image and the second image contain at least partly overlapping road portions;

transform at least one of the second set of 3D lane boundaries and the first set of 3D lane boundaries based on positional data associated with the first image and the second image, such that the first set of 3D lane boundaries and the second set of 3D lane boundaries have a common coordinate system;

evaluate the first set of 3D lane boundaries against the second set of 3D lane boundaries in the common coordinate system in order to find matching lane pairs of the first set of 3D lane boundaries and the second set of 3D lane boundaries; and

update one or more model parameters of the artificial neural network based on a spatio-temporal consistency loss between the found matching lane pairs of the first set of 3D lane boundaries and the second set of 3D lane boundaries.

11. A vehicle comprising:

a camera configured to capture images of at least a portion of a surrounding environment of the vehicle;

a localization system configured to monitor a geographical position of the vehicle; and

an apparatus for training an artificial neural network configured for 3D lane detection based on unlabelled image data from the camera, the apparatus comprising control circuitry configured to:

generate, by means of the artificial neural network, a first set of 3D lane boundaries in a first coordinate system based a first image captured by the vehicle-mounted camera;

generate, by means of the artificial neural network, a second set of 3D lane boundaries in a second coordinate system based on a second image captured by the vehicle mounted camera, wherein the second image is captured at a later moment in time compared to the first image and wherein the first image and the second image contain at least partly overlapping road portions;

transform at least one of the second set of 3D lane boundaries and the first set of 3D lane boundaries based on positional data associated with the first image and the second image, such that the first set of 3D lane boundaries and the second set of 3D lane boundaries have a common coordinate system;

evaluate the first set of 3D lane boundaries against the second set of 3D lane boundaries in the common coordinate system in order to find matching lane pairs of the first set of 3D lane boundaries and the second set of 3D lane boundaries; and

update one or more model parameters of the artificial neural network based on a spatio-temporal consistency loss between the found matching lane pairs of the first set of 3D lane boundaries and the second set of 3D lane boundaries.

12. A remote server comprising the apparatus according to claim 10 .

13. A cloud environment comprising one or more remote servers according to claim 12 .

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (19)