Motion blur and depth of field reconstruction through temporally stable neural networks
A neural network structure, namely a warped external recurrent neural network, is disclosed for reconstructing images with synthesized effects. The effects can include motion blur, depth of field reconstruction (e.g., simulating lens effects), and/or anti-aliasing (e.g., removing artifacts caused by sampling frequency). The warped external recurrent neural network is not recurrent at each layer inside the neural network. Instead, the external state output by the final layer of the neural network is warped and provided as a portion of the input to the neural network for the next image in a sequence of images. In contrast, in a conventional recurrent neural network, hidden state generated at each layer is provided as a feedback input to the generating layer. The neural network can be implemented, at least in part, on a processor. In an embodiment, the neural network is implemented on at least one parallel processing unit.
1 . One or more processors, comprising:
circuitry to:
generate information describing one or more characteristics of one or more blur effects applied by one or more neural networks to generate one or more previous frames of a video; and
use the one or more neural networks to apply at least one of the one or more blur effects to one or more objects in a frame of the video, based at least in part, on one or more inputs of the one or more neural networks comprising the information describing the one or more characteristics of the one or more blur effects used to generate the one or more previous frames of the video.
2 . The one or more processors of claim 1 , wherein the one or more blur effects used to generate the one or more previous frames of the video include at least one of a motion blur effect and a depth of field blur effect.
3 . The one or more processors of claim 1 , wherein the at least one of the one or more blur effects are applied using per-pixel difference data, based on a comparison between the frame and the one or more previous frames of the video.
4 . The one or more processors of claim 1 , wherein the at least one of the one or more blur effects are applied using per-pixel difference data, based on a comparison between the frame and the one or more previous frames of the video, that comprises one or more motion vectors, the one or more motion vectors indicating a relative motion of the one or more objects in the one or more previous frames of the video.
5 . The one or more processors of claim 1 , wherein the one or more blur effects used to generate the one or more previous frames of the video are associated with per-pixel parameters including metadata related to the one or more blur effects used to generate the one or more previous frames of the video.
6 . The one or more processors of claim 5 , wherein the per-pixel parameters include one or more of motion vectors, surface normal vectors, depth values, and depth of field parameters.
7 . The one or more processors of claim 1 , wherein the frame and the one or more previous frames of the video are provided by a graphics processing pipeline configured to render images from one or more three-dimensional models.
8 . A system comprising:
one or more processors to:
generate information describing one or more characteristics of one or more blur effects applied by one or more neural networks to generate one or more previous frames of a video; and
use the one or more neural networks to apply at least one of the one or more blur effects to one or more objects in a frame of the video based, at least in part, on one or more inputs of the one or more neural networks comprising the information describing the one or more characteristics of the one or more blur effects used to generate the one or more previous frames of the video.
9 . The system of claim 8 , wherein the one or more blur effects used to generate the one or more previous frames of the video include at least one of a motion blur effect and a depth of field blur effect.
10 . The system of claim 8 , wherein the at least one of the one or more blur effects are applied using per-pixel difference data, based on a comparison between the frame and the one or more previous frames of the video.
11 . The system of claim 10 , wherein the per-pixel difference data includes one or more motion vectors, the one or more motion vectors indicating a relative motion of the one or more objects in the one or more previous frames of the video.
12 . The system of claim 8 , wherein the one or more blur effects used to generate the one or more previous frames of the video are associated with per-pixel parameters including metadata related to the one or more blur effects used to generate the one or more previous frames of the video, wherein the per-pixel parameters include one or more of motion vectors, surface normal vectors, depth values, and depth of field parameters.
13 . The system of claim 8 , wherein the frame and the one or more previous frames of the video are provided by a graphics processing pipeline configured to render images from one or more three-dimensional models.
14 . A method comprising:
generating information describing one or more characteristics of one or more blur effects applied by one or more neural networks to generate one or more previous frames of a video; and
applying, using the one or more neural networks, at least one of the one or more blur effects to one or more objects in a frame of the video based, at least in part, on one or more inputs of the one or more neural networks comprising the information describing the one or more characteristics of the one or more blur effects used to generate the one or more previous frames of the video.
15 . The method of claim 14 , wherein the one or more blur effects used to generate the one or more previous frames of the video include at least one of a motion blur effect and a depth of field blur effect.
16 . The method of claim 14 , wherein at least one of the one or more blur effects are applied using per-pixel difference data, based on a comparison between the frame and the one or more previous frames of the video.
17 . The method of claim 16 , wherein the per-pixel difference data includes one or more motion vectors, the one or more motion vectors indicating a relative motion of the one or more objects in the one or more previous frames of the video.
18 . The method of claim 14 , wherein the one or more blur effects used to generate the one or more previous frames of the video are associated with per-pixel parameters including metadata related to the one or more blur effects used to generate the one or more previous frames of the video.
19 . The method of claim 18 , wherein the per-pixel parameters include one or more of motion vectors, surface normal vectors, depth values, and depth of field parameters.
20 . The method of claim 14 , wherein the frame and the one or more previous frames of the video are provided by a graphics processing pipeline configured to render images from one or more three-dimensional models.