Reverse rendering of an image based on high definition map data
The autonomous vehicle generates an overlapped image by overlaying HD map data over sensor data and rendering the overlaid images. The visualization process is repeated as the vehicle drives along the route. The visualization may be displayed on a screen within the vehicle or at a remote device. The system performs reverse rendering of a scene based on map data from a selected point. For each line of sight originating at the selected point, the system identifies the farthest object in the map data. Accordingly, the system eliminates objects obstructing the view of the farthest objects in the HD map as viewed from the selected point. The system further allows filtering of objects using filtering criteria based on semantic labels. The system generates a view from the selected point such that 3D objects matching the filtering criteria are eliminated from the view.
1. A non-transitory computer readable storage medium storing instructions encoded thereon that, when executed by a processor, cause the processor to perform steps including:
receiving, by a vehicle, sensor data comprising information about a route of the vehicle;
performing localization based on the sensor data to determine a location of the vehicle;
accessing map data corresponding to a region surrounding the determined location of the vehicle;
identifying a reference point in the map data, the reference point being based on the location of the vehicle;
for each respective angle of a plurality of angles in the region originating at the reference point and being with respect to a ground plane and a corresponding line passing through the reference point, selecting a corresponding point on a first object from the map data, the corresponding point being located from the reference point along the corresponding line, wherein at least some of the angles are associated with a second object located between the reference point and the first object and wherein the second object at least partially obstructs the first object as viewed from the reference point at one or more of the respective viewing angles;
generating an image displaying the selected points from the reference point, wherein the generated image shows the first object and excludes the second object based on the second object at least partially obstructing the first object as viewed from the reference point; and
transmitting the generated image for display on a client device.
2. The non-transitory computer readable storage medium of claim 1 , wherein a height of the reference point is at a threshold height above a ground level that corresponds to the ground plane, the threshold height being greater than a height of objects in the region surrounding the determined location of the vehicle.
3. The non-transitory computer readable storage medium of claim 1 , wherein the stored instructions further cause the processor to perform steps comprising:
overlaying an image of the vehicle on the generated image.
4. The non-transitory computer readable storage medium of claim 1 , wherein the steps are repeated to dynamically update the generated image as the vehicle drives along the route.
5. The non-transitory computer readable storage medium of claim 1 , wherein the generated image shows a perspective view looking down upon the vehicle from the reference point.
6. The non-transitory computer readable storage medium of claim 1 , wherein the stored instructions further cause the processor to perform steps comprising:
moving the reference point as the vehicle moves; and
transmitting the generated image to a remote display device for displaying a video, the video showing the region as observed from the moving reference point as the vehicle moves.
7. The non-transitory computer readable storage medium of claim 1 , wherein the generated image is generated using orthographic projection.
8. The non-transitory computer readable storage medium of claim 1 , wherein the steps are repeated for a plurality of locations of the vehicle to produce a plurality of pre-rendered images.
9. A computer-implemented method comprising:
receiving, by a vehicle, sensor data comprising information about a route of the vehicle;
performing localization based on the sensor data to determine a location of the vehicle;
accessing map data corresponding to a region surrounding the determined location of the vehicle;
identifying a reference point in the map data, the reference point being based on the location of the vehicle;
for each respective angle of a plurality of angles in the region originating at the reference point and being with respect to a ground plane and a corresponding line passing through the reference point, selecting a corresponding point on a first object from the map data, the corresponding point being located from the reference point along the corresponding line, wherein at least some of the angles are associated with a second object located between the reference point and the first object and wherein the second object at least partially obstructs the first object as viewed from the reference point at one or more of the respective viewing angles;
generating an image displaying the selected points from the reference point, wherein the generated image shows the first object and excludes the second object based on the second object at least partially obstructing the first object as viewed from the reference point; and
transmitting the generated image for display on a client device.
10. The computer-implemented method of claim 9 , wherein a height of the reference point is at a threshold height above a ground level that corresponds to the ground plane, the threshold height being greater than a height of objects in the region surrounding the determined location of the vehicle.
11. The computer-implemented method of claim 9 , further comprising:
overlaying an image of the vehicle on the generated image.
12. The computer-implemented method of claim 9 , wherein the steps are repeated to dynamically update the generated image as the vehicle drives along the route.
13. The computer-implemented method of claim 9 , wherein the generated image shows a perspective view looking down upon the vehicle from the reference point.
14. The computer-implemented method of claim 9 , further comprising:
moving the reference point as the vehicle moves; and
transmitting the generated image to a remote display device for displaying a video, the video showing the region as observed from the moving reference point as the vehicle moves.
15. The computer-implemented method of claim 9 , wherein the generated image is generated using orthographic projection.
16. The computer-implemented method of claim 9 , wherein the steps are repeated for a plurality of locations of the vehicle to produce a plurality of pre-rendered images.
17. A computer system comprising:
one or more processors; and
one or more non-transitory computer readable storage media storing instructions encoded thereon that, when executed by the one or more processors a, cause the system to perform steps comprising:
receiving by a vehicle, sensor data comprising information about a route of the vehicle;
performing localization based on the sensor data to determine a location of the vehicle;
accessing map data corresponding to a region surrounding the determined location of the vehicle;
identifying a reference point in the map data, the reference point being based on the location of the vehicle;
for each respective angle of a plurality of angles in the region originating at the reference point and being with respect to a ground plane and a corresponding line passing through the reference point, selecting a corresponding point on a first object from the map data, the corresponding point being located from the reference point along the corresponding line, wherein at least some of the angles are associated with a second object located between the reference point and the first object and wherein the second object at least partially obstructs the first object as viewed from the reference point at one or more of the respective viewing angles;
generating an image displaying the selected points from the reference point, wherein the generated image shows the first object and excludes the second object based on the second object at least partially obstructing the first object as viewed from the reference point; and
transmitting the generated image for display on a client device.
18. The computer system of claim 17 , wherein a height of the reference point is at a threshold height above a ground level that corresponds to the ground plane, the threshold height being greater than a height of objects in the region surrounding the determined location of the vehicle.
19. The computer system of claim 17 , wherein the stored instructions further cause the system to perform steps comprising:
overlaying an image of the vehicle on the generated image.
20. The computer system of claim 17 , wherein the steps are repeated to dynamically update the generated image as the vehicle drives along the route.
21. The computer system of claim 17 , wherein the generated image shows a perspective view looking down upon the vehicle from the reference point.
22. The computer system of claim 17 , wherein the stored instructions further cause the system to perform steps comprising:
moving the reference point as the vehicle moves; and
transmitting the generated image to a remote display device for displaying a video, the video showing the region scene as observed from the moving reference point as the vehicle moves.
23. The computer system of claim 17 , wherein the generated image is generated using orthographic projection.
24. The computer system of claim 17 , wherein the steps are repeated for a plurality of locations of the vehicle to produce a plurality of pre-rendered images.
25. The non-transitory computer readable storage medium of claim 1 , wherein the reference point is located at a height above the ground plane.