Parametric top-view representation of complex road scenes

1. A method for producing a road layout model, comprising:

capturing a plurality of sequential digital images using a video camera, wherein the digital images are of a perspective view;

converting each of the plurality of sequential digital images into a top-down view image using a processor through semantic segmentation, wherein each object in the sequential digital images is segmented by clustering pixels of the object into ground truth classes, and assigning a depth value to each pixel;

conveying a top-down view image of time t to a neural network that performs a feature transform on the top-down view image of time t to form a feature map of time t;

transferring the feature map of the top-down view image of time t to a feature transform module configured to warp the feature map of time t to a time t+1;

conveying a top-down view image of time t+1 to the neural network that performs a feature transform on the top-down view image of time t+1 to form a feature map of time t+1;

combining the warped feature map of time t with the feature map of time t+1 to form a combined feature map;

transferring the combined feature map to a long short-term memory (LSTM) module to generate the road layout model; and

displaying the road layout model to a user.

2. The method as recited in claim 1 , wherein the road layout model is displayed to a user in a vehicle to make driving decisions in real time.

3. The method as recited in claim 1 , wherein the feature maps are combined using a simple weighted summation, f com (F t , g ftm (F t−1 ))=α·F t +(1−α)·g ftm (F t−1 ), wherein f com represents a combined feature map, F t is the feature map of time t, F t−1 is the feature map of time t−1, and g ftm is result of the feature transform module.

4. The method as recited in claim 1 , further comprising converting the combined feature map into a 1-dimensional feature vector for each frame before transferring the 1-dimensional feature vector to the long short-term memory (LSTM) module.

5. The method as recited in claim 1 , wherein each of the plurality of sequential digital images is converted into a plurality of top-down view images by back-projecting all road pixels into a 3D point cloud and projecting all 3D points onto an x-y plane.

6. The method as recited in claim 1 , further comprising conveying an output from the long short-term memory (LSTM) module into a multi-layer perceptron (MLP).

7. A processing system for producing a road layout model, comprising:

one or more processor devices;

a memory in communication with at least one of the one or more processor devices; and

a display screen;

wherein the processing system includes a graphical modeler configured to receive as input a plurality of sequential digital images, wherein the digital images are of a perspective view, convert each of the plurality of sequential digital images into a top-down view image using a processor through semantic segmentation, wherein each object in the sequential digital images is segmented by clustering pixels of the object into ground truth classes, and assigning a depth value to each pixel, convey a top-down view image of time t to a neural network that performs a feature transform on the top-down view image of time t to form a feature map of time t, and convey a top-down view image of time t+1 to the neural network that performs a feature transform on the top-down view image of time t+1 to form a feature map of time t+1;

a feature transform module configured to receive the feature map of the top-down view image of time t and warp the feature map of time t to a time t+1;

a combiner configured to combine the warped feature map of time t with the feature map of time t+1 to form a combined feature map;

a long short-term memory (LSTM) module configured to receive the combined feature map and to generate the road layout model; and

the processor device is configured to transmit the road layout model to the display screen for presentation to a user.

8. The processing system as recited in claim 7 , wherein the road layout model is displayed to a user in a vehicle to make driving decisions in real time.

9. The processing system as recited in claim 7 , wherein the feature maps are combined using a simple weighted summation, f com (F t , g ftm (F t−1 ))=α·F t +(1−α)·g ftm (F t−1 ), wherein f rom represents a combined feature map, F t is the feature map of time t, F t−1 is the feature map of time t−1, and g ftm is result of the feature transform module.

10. The processing system as recited in claim 7 , further comprising converting the combined feature map into a 1-dimensional feature vector for each frame before transferring the 1-dimensional feature vector to the long short-term memory (LSTM) module.

11. The processing system as recited in claim 7 , wherein each of the plurality of sequential digital images is converted into a top-down view image by back-projecting all road pixels into a 3D point cloud and projecting all 3D points onto an x-y plane.

12. The processing system as recited in claim 7 , further comprising conveying an output from the long short-term memory (LSTM) module into a multi-layer perceptron (MLP).

13. A non-transitory computer readable storage medium comprising a computer readable program for producing a road layout model, wherein the computer readable program when executed on a computer causes the computer to perform the steps of:

capturing a plurality of sequential digital images using a video camera, wherein the digital images are of a perspective view;

converting each of the plurality of sequential digital images to a top-down view image using a processor through semantic segmentation, wherein each object in the sequential digital images is segmented by clustering pixels of the object into ground truth classes, and assigning a depth value to each pixel;

conveying a top-down view image of time t to a neural network that performs a feature transform on the top-down view image of time t to form a feature map of time t;

transferring the feature map of the top-down view image of time t into a feature transform module configured to warp the feature map of time t to a time t+1;

conveying a top-down view image of time t+1 to the neural network that performs a feature transform on the top-down view image of time t+1 to form a feature map of time t+1;

combining the warped feature map of time t with the feature map of time t+1 to form a combined feature map;

transferring the combined feature map to a long short-term memory (LSTM) module to generate the road layout model; and

displaying the road layout model to a user.

14. The computer readable program as recited in claim 13 , wherein the road layout model is displayed to a user in a vehicle to make driving decisions in real time.

15. The computer readable program as recited in claim 13 , wherein the feature maps are combined using a simple weighted summation, f com (F t , g ftm (F t−1 ))=α·F t +(1−α)·g ftm (F t−1 ), wherein f com represents a combined feature map, F t is the feature map of time t, F t−1 is the feature map of time t−1, and g ftm is result of the feature transform module.

16. The computer readable program as recited in claim 13 , further comprising converting the combined feature map into a 1-dimensional feature vector for each frame before transferring the 1-dimensional feature vector to the long short-term memory (LSTM) module.

17. The computer readable program as recited in claim 13 , wherein each of the plurality of sequential digital images is converted into a top-down view image by back-projecting all road pixels into a 3D point cloud and projecting all 3D points onto an x-y plane.

18. The computer readable program as recited in claim 13 , further comprising conveying an output from the long short-term memory (LSTM) module into a multi-layer perceptron (MLP).