LANE DETECTION AND DISTANCE ESTIMATION USING SINGLE-VIEW GEOMETRY
Disclosed are methods, devices, and computer-readable media for detecting lanes and objects in image frames of a monocular camera. In one embodiment, a method is disclosed comprising receiving a sample set of image frames; detecting a plurality of markers in the sample set of image frames using a convolutional neural network (CNN); fitting lines based on the plurality of markers; detecting a plurality of vanishing points based on the lines; identifying a best fitting horizon for the sample set of image frames via a RANSAC algorithm; computing an inverse perspective mapping (IPM) based on the best fitting horizon; and computing a lane width estimate based on the sample set of image frames using the IPM in a rectified view and the parallel line fitting.
1 - 20 . (canceled)
21 . A method comprising:
receiving video frames captured by a camera installed on a vehicle;
detecting markers in the video frames using a machine learning model;
fitting one or more lines to the markers;
determining if the camera is being initialized; and
when the camera is not being initialized:
retrieving a camera height and a road plane normal,
reconstructing a road plane using the camera height and the road plane normal,
detecting an object in the video frames,
determining an intersection of a point in the image plane with the reconstructed road plane, and
calculating a distance to the object based on the intersection.
22 . The method of claim 21 , wherein calculating the distance is performed in real-time or near real-time on one of an onboard device or a remote computing environment.
23 . The method of claim 21 , wherein deep learning-based lane line detection algorithms are executed upon initialization and are not executed after initialization.
24 . The method of claim 21 , further comprising computing the camera height during initialization by:
calculating a lane width in pixels from the video frames;
computing a viewing angle based on the lane width in pixels and a focal length of the camera; and
computing the camera height using the viewing angle and a real world lane width.
25 . The method of claim 21 , further comprising:
rectifying lane boundary markers detected in the video frames using an inverse perspective mapping;
fitting lines to the lane boundary markers;
calculating a lane boundary offset for missing or low confidence lanes using an initialized lane width in pixels; and
predicting missing lane boundaries by adding the lane boundary offset to an x-intercept of detected lane boundaries in a rectified view.
26 . The method of claim 25 , further comprising performing a reciprocal weighted average of x-coordinates of the lines, wherein weights are selected based on the distance of a predicted lane boundary from a detected lane boundary, with a detected lane boundary given a first weight, a predicted lane boundary that is one lane width distance away assigned a second weight, and a predicted lane boundary that is two lane widths away assigned a third weight.
27 . The method of claim 21 , wherein determining if the camera is being initialized comprises:
identifying three co-linear points in an image that intersect three lane lines;
determining that a ratio of line segments is equal to a ratio of distances between the three lane lines; and
identifying a lateral vanishing point to estimate a horizon line.
28 . A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining steps of:
receiving video frames captured by a camera installed on a vehicle;
detecting markers in the video frames using a machine learning model;
fitting one or more lines to the markers;
determining if the camera is being initialized; and
when the camera is not being initialized:
retrieving a camera height and a road plane normal,
reconstructing a road plane using the camera height and the road plane normal,
detecting an object in the video frames,
determining an intersection of a point in the image plane with the reconstructed road plane, and
calculating a distance to the object based on the intersection.
29 . The non-transitory computer-readable storage medium of claim 28 , wherein calculating the distance is performed in real-time or near real-time on one of an onboard device or a remote computing environment.
30 . The non-transitory computer-readable storage medium of claim 28 , wherein deep learning-based lane line detection algorithms are executed upon initialization and are not executed after initialization.
31 . The non-transitory computer-readable storage medium of claim 28 , the steps further comprising computing the camera height during initialization by:
calculating a lane width in pixels from the video frames;
computing a viewing angle based on the lane width in pixels and a focal length of the camera; and
computing the camera height using the viewing angle and a real world lane width.
32 . The non-transitory computer-readable storage medium of claim 28 , the steps further comprising:
rectifying lane boundary markers detected in the video frames using an inverse perspective mapping;
fitting lines to the lane boundary markers;
calculating a lane boundary offset for missing or low confidence lanes using an initialized lane width in pixels; and
predicting missing lane boundaries by adding the lane boundary offset to an x-intercept of detected lane boundaries in a rectified view.
33 . The non-transitory computer-readable storage medium of claim 32 , the steps further comprising performing a reciprocal weighted average of x-coordinates of the lines, wherein weights are selected based on the distance of a predicted lane boundary from a detected lane boundary, with a detected lane boundary given a first weight, a predicted lane boundary that is one lane width distance away assigned a second weight, and a predicted lane boundary that is two lane widths away assigned a third weight.
34 . The non-transitory computer-readable storage medium of claim 28 , wherein determining if the camera is being initialized comprises:
identifying three co-linear points in an image that intersect three lane lines;
determining that a ratio of line segments is equal to a ratio of distances between the three lane lines; and
identifying a lateral vanishing point to estimate a horizon line.
35 . A device comprising:
a processor; and
a storage medium for tangibly storing thereon program logic for execution by the processor, the program logic comprising steps for:
receiving video frames captured by a camera installed on a vehicle;
detecting markers in the video frames using a machine learning model;
fitting one or more lines to the markers;
determining if the camera is being initialized; and
when the camera is not being initialized:
retrieving a camera height and a road plane normal,
reconstructing a road plane using the camera height and the road plane normal,
detecting an object in the video frames,
determining an intersection of a point in the image plane with the reconstructed road plane, and
calculating a distance to the object based on the intersection.
36 . The device of claim 35 , wherein deep learning-based lane line detection algorithms are executed upon initialization and are not executed after initialization.
37 . The device of claim 35 , the steps further comprising computing the camera height during initialization by:
calculating a lane width in pixels from the video frames;
computing a viewing angle based on the lane width in pixels and a focal length of the camera; and
computing the camera height using the viewing angle and a real world lane width.
38 . The device of claim 35 , the steps further comprising:
rectifying lane boundary markers detected in the video frames using an inverse perspective mapping;
fitting lines to the lane boundary markers;
calculating a lane boundary offset for missing or low confidence lanes using an initialized lane width in pixels; and
predicting missing lane boundaries by adding the lane boundary offset to an x-intercept of detected lane boundaries in a rectified view.
39 . The device of claim 38 , the steps further comprising performing a reciprocal weighted average of x-coordinates of the lines, wherein weights are selected based on the distance of a predicted lane boundary from a detected lane boundary, with a detected lane boundary given a first weight, a predicted lane boundary that is one lane width distance away assigned a second weight, and a predicted lane boundary that is two lane widths away assigned a third weight.
40 . The device of claim 35 , wherein determining if the camera is being initialized comprises:
identifying three co-linear points in an image that intersect three lane lines;
determining that a ratio of line segments is equal to a ratio of distances between the three lane lines; and
identifying a lateral vanishing point to estimate a horizon line.