Neural reconstruction of sequential frames
In one embodiment, a computing system configured to generate a current frame may access a current sample dataset having incomplete pixel information of a current frame in a sequence of frames. The system may access a previous frame in the sequence of frames with complete pixel information. The system may further access a motion representation indicating pixel relationships between the current frame and the previous frame. The previous frame may then be transformed according to the motion representation. The system may generate the current frame having complete pixel information by processing the current sample dataset and the transformed previous frame using a first machine-learning model.
1. A method comprising, by a computing system:
accessing a current sample dataset having incomplete pixel information of a current frame in a sequence of frames, wherein the current sample dataset is generated based on a corresponding binary mask, wherein the corresponding binary mask represents whether color information for each pixel in the current frame is sampled;
accessing at least one previous frame in the sequence of frames with complete pixel information;
accessing a motion representation indicating pixel relationships between the current frame and the previous frame;
transforming the previous frame according to the motion representation;
accessing a first machine-learning model;
providing the current sample dataset, the corresponding binary mask, and the transformed previous frame as inputs to the first machine-learning model; and
generating the current frame by processing the current sample dataset and the transformed previous frame using the first machine-learning model, wherein the generated current frame has complete pixel information.
2. The method of claim 1 , wherein the complete pixel information of the generated current frame includes the incomplete pixel information of the current sample dataset and additional pixel information generated by the first machine-learning model.
3. The method of claim 1 , wherein the incomplete pixel information of the current sample dataset is generated by a rendering system.
4. The method of claim 1 , wherein:
the incomplete pixel information of the current sample dataset includes a first region and a second region;
the first region has denser pixel information than the second region; and
the first region corresponds to a foveal region of a user and the second region is outside of the foveal region.
5. The method of claim 1 , wherein the previous frame with complete pixel information is generated using the first machine-learning model and a previous sample dataset having incomplete pixel information of the previous frame.
6. The method of claim 1 , wherein the motion representation maps one or more first pixel locations in the previous frame to one or more second pixel locations in the transformed previous frame.
7. The method of claim 1 , wherein:
the motion representation is generated based on visibility tests performed by a rendering system for the current frame and the previous frame.
8. The method of claim 1 , further comprising:
generating the motion representation by processing an incomplete motion representation using a second machine-learning model.
9. The method of claim 1 , wherein
the motion representation is generated by processing an incomplete motion representation using the first machine-learning model.
10. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
access a current sample dataset having incomplete pixel information of a current frame in a sequence of frames, wherein the current sample dataset is generated based on a corresponding binary mask, wherein the corresponding binary mask represents whether color information for each pixel in the current frame is sampled;
access at least one previous frame m the sequence of frames with complete pixel information;
access a motion representation indicating pixel relationships between the current frame and the previous frame;
transform the previous frame according to the motion representation;
access a first machine-learning model;
provide the current sample dataset, the corresponding binary mask, and the transformed previous frame as inputs to the first machine-learning model; and
generate the current frame by processing the current sample dataset and the transformed previous frame using the first machine-learning model, wherein the generated current frame has complete pixel information.
11. The media of claim 10 , wherein the complete pixel information of the generated current frame includes the incomplete pixel information of the current sample dataset and additional pixel information generated by the first machine-learning model.
12. The media of claim 10 , wherein the incomplete pixel information of the current sample dataset is generated by a rendering system.
13. The media of claim 10 , wherein:
the incomplete pixel information of the current sample dataset includes a first region and a second region;
the first region has denser pixel information than the second region; and
the first region corresponds to a foveal region of a user and the second region is outside of the foveal region.
14. The media of claim 10 , wherein the previous frame with complete pixel information is generated using the first machine-learning model and a previous sample dataset having incomplete pixel information of the previous frame.
15. The media of claim 10 , wherein the motion representation maps one or more first pixel locations in the previous frame to one or more second pixel locations in the transformed previous frame.
16. A system comprising: one or more processors; and one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:
access a current sample dataset having incomplete pixel information of a current frame in a sequence of frames, wherein the current sample dataset is generated based on a corresponding binary mask, wherein the corresponding binary mask represents whether color information for each pixel in the current frame is sampled;
access at least one previous frame m the sequence of frames with complete pixel information;
access a motion representation indicating pixel relationships between the current frame and the previous frame;
transform the previous frame according to the motion representation;
access a first machine-learning model;
provide the current sample dataset, the corresponding binary mask, and the transformed previous frame as inputs to the first machine-learning model; and
generate the current frame by processing the current sample dataset and the transformed previous frame using the first machine-learning model, wherein the generated current frame has complete pixel information.
17. The system of claim 16 , wherein the complete pixel information of the generated current frame includes the incomplete pixel information of the current sample dataset and additional pixel information generated by the first machine-learning model.
18. The system of claim 16 , wherein the incomplete pixel information of the current sample dataset is generated by a rendering system.
19. The system of claim 16 , wherein:
the incomplete pixel information of the current sample dataset includes a first region and a second region;
the first region has denser pixel information than the second region; and
the first region corresponds to a foveal region of a user and the second region is outside of the foveal region.
20. The system of claim 16 , wherein the previous frame with complete pixel information is generated using the first machine-learning model and a previous sample dataset having incomplete pixel information of the previous frame.