Rendering Video Of A Scene Using Three-Dimensional Gaussians
A set of images of a scene re received. Each image includes temporal data and spatial data relating to the scene. Based on the spatial data of each image, three-dimensional (3D) Gaussian splatting data is generated. The temporal data of each image and the 3D Gaussian splatting data are inputted to a neural network to generate spatial-temporal 3D Gaussian embeddings. Offset data based on the spatial-temporal 3D Gaussian embeddings is generated. The video of the scene is rendered based on the 3D Gaussian splatting data and the offset data, allowing for improved rendering of video of the scene.
1 . A method of rendering video of a scene, comprising:
receiving a set of images of the scene, wherein each image comprises temporal data and spatial data relating to the scene;
generating, based on the spatial data of each image, three-dimensional (3D) Gaussian splatting data;
inputting the temporal data of each image and the 3D Gaussian splatting data to a neural network to generate spatial-temporal 3D Gaussian embeddings;
generating offset data based on the spatial-temporal 3D Gaussian embeddings; and
rendering the video of the scene based on the 3D Gaussian splatting data and the offset data.
2 . The method of claim 1 , wherein rendering the video of the scene comprises:
combining the 3D Gaussian splatting data with the offset data to generate spatial-temporal 3D Gaussian representations of the scene; and
rendering the video of the scene by inputting the spatial-temporal 3D Gaussian representations to a rasterizer.
3 . The method of claim 1 , wherein generating the 3D Gaussian splatting data comprises generating the 3D Gaussian splatting data using 3D point cloud reconstruction.
4 . The method of claim 1 , wherein generating the 3D Gaussian splatting data comprises inputting the spatial data of each image to a machine learning model trained to generate 3D Gaussian splatting data based on spatial data from one or more images.
5 . The method of claim 1 , wherein:
each image further comprises viewpoint data of the scene; and
generating the offset data is further based on the viewpoint data.
6 . The method of claim 5 , wherein generating the offset data comprises inputting the viewpoint data to a neural network to generate one or more spherical harmonics offset parameters.
7 . The method of claim 1 , wherein the neural network is a multi-layer perceptron.
8 . The method of claim 1 , wherein generating the offset data comprises inputting the spatial-temporal 3D Gaussian embeddings to one or more neural networks.
9 . The method of claim 8 , wherein at least one of the one or more neural networks is a multi-layer perceptron.
10 . The method of claim 8 , wherein each of the one or more neural networks is a multi-layer perceptron.
11 . The method of claim 1 , wherein:
generating the 3D Gaussian splatting data comprises generating one or more of: one or more 3D Gaussian position parameters; one or more 3D Gaussian scale parameters; one or more 3D Gaussian rotation parameters; and one or more 3D Gaussian opacity parameters; and
generating the offset data comprises inputting one or more of:
the one or more 3D Gaussian position parameters to a neural network to generate one or more position offset parameters;
the one or more 3D Gaussian scale parameters to a neural network to generate one or more scale offset parameters;
the one or more 3D Gaussian rotation parameters to a neural network to generate one or more rotation offset parameters; and
the one or more 3D Gaussian opacity parameters to a neural network to generate one or more opacity offset parameters.
12 . The method of claim 1 , wherein generating the 3D Gaussian splatting data comprises:
identifying, within the spatial data of each image:
foreground spatial data relating a foreground of the scene; and
background spatial data relating a background of the scene;
generating, based on the background spatial data, background 3D Gaussian splatting data; and
generating, based on the foreground spatial data, foreground 3D Gaussian splatting data.
13 . The method of claim 12 , wherein generating the spatial-temporal 3D Gaussian embeddings comprises:
generating background spatial-temporal 3D Gaussian embeddings based on the temporal data of each image and the background 3D Gaussian splatting data; and
generating foreground spatial-temporal 3D Gaussian embeddings based on the temporal data of each image and the foreground 3D Gaussian splatting data.
14 . The method of claim 13 , wherein generating the offset data comprises:
generating background offset data based on the background spatial-temporal 3D Gaussian embeddings; and
generating foreground offset data based on the foreground spatial-temporal 3D Gaussian embeddings.
15 . The method of claim 14 , wherein rendering the video of the scene comprises:
combining the background 3D Gaussian splatting data with the background offset data to generate spatial-temporal 3D Gaussian representations of the background of the scene;
combining the foreground 3D Gaussian splatting data with the foreground offset data to generate spatial-temporal 3D Gaussian representations of the foreground of the scene; and
rendering the video of the scene by inputting the spatial-temporal 3D Gaussian representations of the background and the foreground of the scene to the rasterizer.
16 . The method of claim 14 , wherein:
generating the background offset data comprises inputting the background spatial-temporal 3D Gaussian embeddings to a single neural network to generate the background offset data; and
generating the foreground offset data comprises inputting the foreground spatial-temporal 3D Gaussian embeddings to a single neural network to generate the foreground offset data.
17 . The method of claim 1 , wherein generating the offset data comprises inputting the spatial-temporal 3D Gaussian embeddings to a single neural network to generate the offset data.
18 . A non-transitory, computer-readable medium storing computer program code configured, when executed by one or more processors, to cause the one or more processors to perform a method comprising:
receiving a set of images of a scene, wherein each image comprises temporal data and spatial data relating to the scene;
generating, based on the spatial data of each image, three-dimensional (3D) Gaussian splatting data;
inputting the temporal data of each image and the 3D Gaussian splatting data to a neural network to generate spatial-temporal 3D Gaussian embeddings;
generating offset data based on the spatial-temporal 3D Gaussian embeddings; and
rendering the video of the scene based on the 3D Gaussian splatting data and the offset data.
19 . A computing device comprising one or more graphics processors operable to render video of a scene by:
receiving a set of images of a scene, wherein each image comprises temporal data and spatial data relating to the scene;
generating, based on the spatial data of each image, three-dimensional (3D) Gaussian splatting data;
inputting the temporal data of each image and the 3D Gaussian splatting data to a neural network to generate spatial-temporal 3D Gaussian embeddings;
generating offset data based on the spatial-temporal 3D Gaussian embeddings; and
rendering the video of the scene based on the 3D Gaussian splatting data and the offset data.