Latency-resilient cloud rendering
A method includes the steps of receiving training data comprising images of an object and associated camera poses from which the images are captured, training, based on the training data, a machine-learning model to take as input a given viewpoint and synthesize an image of a virtual representation of the object viewed from the given viewpoint, generating, for each of predetermined viewpoints surrounding the virtual representation of the object, a view-dependent image of the object as viewed from that viewpoint using the trained machine-learning model, receiving, from a client device, a desired viewpoint from which to view the virtual representation of the object, selecting one or more of the predetermined viewpoints based on the desired viewpoint, and sending, to the client device, the view-dependent images associated with the selected one or more viewpoints for rendering an output image of the virtual representation of the object viewed from the desired viewpoint.
1. A method comprising, by a computing server:
receiving training data comprising images of an object and associated camera poses from which the images are captured;
training, based on the training data, a machine-learning model to take as input a given viewpoint and synthesize an image of a virtual representation of the object as viewed from the given viewpoint;
querying the trained machine-learning model with a predetermined plurality of viewpoints surrounding the virtual representation of the object;
generating, for each of the predetermined plurality of viewpoints surrounding the virtual representation of the object, a view-dependent image of the object as viewed from that viewpoint using the trained machine-learning model;
receiving, from a client device, a desired viewpoint from which to view the virtual representation of the object;
selecting three or more of the predetermined plurality of viewpoints based on that are surrounding the desired viewpoint; and
sending, to the client device, the view-dependent images associated with the selected three or more viewpoints for rendering an output image of the virtual representation of the object viewed from the desired viewpoint.
2. The method of claim 1 , wherein the machine-learning model is a neural network model trained to learn a neural radiance field of the object.
3. The method of claim 1 , wherein each of the generated view-dependent images represents an image of the virtual representation of the object as viewed from a coordinate point (x, y, z) with a view direction (θ,Φ).
4. The method of claim 1 , wherein selecting the three or more of the predetermined plurality of viewpoints based on the desired viewpoint comprises:
prioritizing three or more of the predetermined plurality of viewpoints based on distances between the desired viewpoint and each of the predetermined plurality of viewpoints; and
selecting the three or more viewpoints from the predetermined plurality of viewpoints based on their prioritization.
5. The method of claim 4 , wherein prioritizing the three or more of the predetermined plurality of viewpoints is further based on a trajectory of a virtual camera associated with the desired viewpoint.
6. The method of claim 1 , wherein rendering the output image of the virtual representation of the object as viewed from the desired viewpoint comprises:
reprojecting the view-dependent images associated with the selected three or more viewpoints from the desired viewpoint.
7. The method of claim 1 , wherein generating, for each of the predetermined plurality of viewpoints surrounding the virtual representation of the object, the view-dependent image of the object as viewed from that viewpoint using the trained machine-learning model is performed prior to receiving, from the client device, the desired viewpoint from which to view the virtual representation of the object.
8. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
receive training data comprising images of an object and associated camera poses from which the images are captured;
train, based on the training data, a machine-learning model to take as input a given viewpoint and synthesize an image of a virtual representation of the object as viewed from the given viewpoint;
query the trained machine-learning model with a predetermined plurality of viewpoints surrounding the virtual representation of the object
generate, for each of the predetermined plurality of viewpoints surrounding the virtual representation of the object, a view-dependent image of the object as viewed from that viewpoint using the trained machine-learning model;
receive, from a client device, a desired viewpoint from which to view the virtual representation of the object;
select three or more of the predetermined plurality of viewpoints based on that are surrounding the desired viewpoint; and
send, to the client device, the view-dependent images associated with the selected three or more viewpoints for rendering an output image of the virtual representation of the object viewed from the desired viewpoint.
9. The one or more computer-readable non-transitory storage media of claim 8 , wherein the machine-learning model is a neural network model trained to learn a neural radiance field of the object.
10. The one or more computer-readable non-transitory storage media of claim 8 , wherein each of the generated view-dependent images represents an image of the virtual representation of the object as viewed from a coordinate point (x, y, z) with a view direction (θ,Φ).
11. The one or more computer-readable non-transitory storage media of claim 8 , wherein selecting the three or more of the predetermined plurality of viewpoints based on the desired viewpoint comprises:
prioritizing three or more of the predetermined plurality of viewpoints based on distances between the desired viewpoint and each of the predetermined plurality of viewpoints; and
selecting the three or more viewpoints from the predetermined plurality of viewpoints based on their prioritization.
12. The one or more computer-readable non-transitory storage media of claim 11 , wherein prioritizing the three or more of the predetermined plurality of viewpoints is further based on a trajectory of a virtual camera associated with the desired viewpoint.
13. The one or more computer-readable non-transitory storage media of claim 8 , wherein rendering the output image of the virtual representation of the object as viewed from the desired viewpoint comprises:
reprojecting the view-dependent images associated with the selected three or more viewpoints from the desired viewpoint.
14. The one or more computer-readable non-transitory storage media of claim 8 , wherein generating, for each of the predetermined plurality of viewpoints surrounding the virtual representation of the object, the view-dependent image of the object as viewed from that viewpoint using the trained machine-learning model is performed prior to receiving, from the client device, the desired viewpoint from which to view the virtual representation of the object.
15. A system comprising: one or more processors; and one or more computer-readable non-transitory storage media in communication with the one or more processors, the one or more computer-readable non-transitory storage media comprising instructions that when executed by the one or more processors, cause the system to perform:
receive training data comprising images of an object and associated camera poses from which the images are captured;
train, based on the training data, a machine-learning model to take as input a given viewpoint and synthesize an image of a virtual representation of the object as viewed from the given viewpoint;
query the trained machine-learning model with a predetermined plurality of viewpoints surrounding the virtual representation of the object
generate, for each of the predetermined plurality of viewpoints surrounding the virtual representation of the object, a view-dependent image of the object as viewed from that viewpoint using the trained machine-learning model;
receive, from a client device, a desired viewpoint from which to view the virtual representation of the object;
select three or more of the predetermined plurality of viewpoints based on that are surrounding the desired viewpoint; and
send, to the client device, the view-dependent images associated with the selected three or more viewpoints for rendering an output image of the virtual representation of the object viewed from the desired viewpoint.
16. The system of claim 15 , wherein the machine-learning model is a neural network model trained to learn a neural radiance field of the object.
17. The system of claim 15 , wherein each of the generated view-dependent images represents an image of the virtual representation of the object as viewed from a coordinate point (x, y, z) with a view direction (θ,Φ).
18. The system of claim 15 , wherein selecting the three or more of the predetermined plurality of viewpoints based on the desired viewpoint comprises:
prioritizing three or more of the predetermined plurality of viewpoints based on distances between the desired viewpoint and each of the predetermined plurality of viewpoints; and
selecting the three or more viewpoints from the predetermined plurality of viewpoints based on their prioritization.
19. The system of claim 18 , wherein prioritizing the three or more of the predetermined plurality of viewpoints is further based on a trajectory of a virtual camera associated with the desired viewpoint.
20. The system of claim 15 , wherein rendering the output image of the virtual representation of the object as viewed from the desired viewpoint comprises:
reprojecting the view-dependent images associated with the selected three or more viewpoints from the desired viewpoint.