Synthesizing three-dimensional visualizations from perspectives of onboard sensors of autonomous vehicles

1. A method of generating a visualization of a three-dimensional (3D) world view from a perspective of a camera of a vehicle, the method comprising:

receiving, by one or more processors, images of a scene captured by the camera of the vehicle and 3D content for the scene;

identifying, by the one or more processors, a virtual camera model for the camera of the vehicle based on parameters associated with the camera of the vehicle;

generating, by the one or more processors, a set of matrices using the identified virtual camera model;

applying, by the one or more processors, the set of matrices to the 3D content to create the 3D world view; and

generating the visualization, by the one or more processors, using the 3D world view as an overlay with an image of the images, wherein the visualization provides a real-world image from the perspective of the camera of the vehicle with one or more graphical overlays of the 3D content.

2. The method of claim 1 , wherein the parameters include intrinsic and extrinsic parameters of the camera of the vehicle.

3. The method of claim 2 , wherein the parameters include focal length, aperture size, position and rotation of the camera relative to the vehicle when the images were captured.

4. The method of claim 1 , wherein the set of matrices includes a view matrix.

5. The method of claim 4 , wherein the view matrix transforms from world-space coordinates into camera-space coordinates.

6. The method of claim 4 , wherein applying the set of matrices includes generating coordinates for the 3D content in a reference frame of the camera of the vehicle.

7. The method of claim 4 , wherein the view matrix is defined as a combination of a swap matrix, a camera extrinsics matrix, and a vehicle pose matrix.

8. The method of claim 7 , wherein the swap matrix is configured to perform a coordinate swap to transform from vehicle coordinates to 3D rendering coordinates.

9. The method of claim 7 , wherein the camera extrinsics matrix provides a 3D position and 3D rotation of the camera of the vehicle when the image of the images was captured.

10. The method of claim 7 , wherein the vehicle pose matrix provides rotation and position of the vehicle when the image of the images was captured.

11. The method of claim 1 , wherein the set of matrices includes a projection matrix.

12. The method of claim 11 , wherein the projection matrix is configured to transform 3D coordinates in a reference frame of the camera of the vehicle into clip-space coordinates.

13. The method of claim 11 , wherein the projection matrix is defined as a combination of a transformation matrix and an orthographic projection matrix.

14. The method of claim 13 , wherein the transformation matrix is configured to generate projected coordinates using a linear transformation of 3D content and distorting points from a perspective of the camera of the vehicle.

15. The method of claim 14 , wherein the orthographic projection matrix is configured to convert the projected coordinates into clip space coordinates.

16. The method of claim 1 , wherein the 3D world view incorporates 3D content from space outside of the identified virtual camera model.

17. The method of claim 1 , wherein the 3D content includes 3D bounding boxes for objects.

18. The method of claim 1 , wherein the 3D content includes one or more of 3D LIDAR points, 3D LIDAR point clouds, or 3D surfel maps.

19. The method of claim 1 , wherein the 3D content includes features of a roadgraph.

20. The method of claim 1 , wherein the 3D content includes behavior prediction trajectories for objects.

21. The method of claim 1 , wherein the vehicle includes a plurality of cameras, the method further comprising:

identifying, by one or more processors, a virtual camera model for each camera of the plurality the cameras; and

generating, by the one or more processors, a set of matrices for each camera of the plurality of cameras of the vehicle using the virtual camera model for the camera.

22. The method of claim 1 , wherein the set of matrices includes a view matrix that transforms from world-space coordinates into camera-space coordinates, the set of matrices including a projection matrix that transforms 3D coordinates in a reference frame of the camera of the vehicle.

23. A method of generating a visualization of a three-dimensional (3D) world view from a perspective of a camera of a vehicle, the method comprising:

receiving, by one or more processors, images of a scene captured by the camera of the vehicle and 3D content for the scene;

identifying, by the one or more processors, a virtual camera model for the camera;

generating, by the one or more processors, a set of matrices using the virtual camera model, wherein the set of matrices includes a view matrix that transforms from world-space coordinates into camera-space coordinates, the set of matrices including a projection matrix that transforms 3D coordinates in a reference frame of the camera of the vehicle;

applying, by the one or more processors, the set of matrices to the 3D content to create the 3D world view; and

generating the visualization, by the one or more processors, using the 3D world view as an overlay with an image of the images, wherein the visualization provides a real-world image from the perspective of the camera of the vehicle with one or more graphical overlays of the 3D content.