Switchable propagation neural network
A temporal propagation network (TPN) system learns the affinity matrix for video image processing tasks. An affinity matrix is a generic matrix that defines the similarity of two points in space. The TPN system includes a guidance neural network model and a temporal propagation module and is trained for a particular computer vision task to propagate visual properties from a key-frame represented by dense data (color), to another frame that is represented by coarse data (grey-scale). The guidance neural network model generates an affinity matrix referred to as a global transformation matrix from task-specific data for the key-frame and the other frame. The temporal propagation module applies the global transformation matrix to the key-frame property data to produce propagated property data (color) for the other frame. For example, the TPN system may be used to colorize several frames of greyscale video using a single manually colorized key-frame.
1. A computer-implemented method, comprising:
receiving task-specific data for a key-frame of a video sequence defining attributes of pixels in the key-frame;
receiving property data for the pixels in the key-frame;
receiving task-specific data for a frame of the video sequence defining attributes of pixels in the frame;
processing, according to parameters, by a guidance neural network model, the task-specific data for a key-frame and the task-specific data for the frame to produce guidance data for a task, wherein the guidance data comprises at least two affinity values that are aligned in either the horizontal or vertical pixel dimension of the key-frame for transitions from the key-frame to the frame; and
applying the guidance data to the property data for the pixels in the key-frame to generate property data for the pixels in the frame, wherein the guidance data preserves a style energy of the property data for the key-frame in the generated property data for the frame.
2. The computer-implemented method of claim 1 , wherein the guidance data further comprises task-specific affinity values for transitions from the key-frame to other frames in the video sequence.
3. The computer-implemented method of claim 1 , wherein the guidance neural network model is trained during generation of the property data for the frame using a training dataset for the task.
4. The computer-implemented method of claim 1 , wherein the task is colorization, the attributes are lightness data, the property data for the key-frame is color data corresponding to the key-frame and the property data for the frame comprises color data corresponding to the frame.
5. The computer-implemented method of claim 1 , wherein the task is segmentation, the attributes are color data, the property data for the key-frame is segmentation data corresponding to the key-frame and the property data for the frame comprises segmentation data corresponding to the frame.
6. A computer-implemented method, comprising:
receiving task-specific data for a key-frame of a video sequence defining attributes of pixels in the key-frame;
receiving task-specific data for a frame of the video sequence defining attributes of pixels in the frame;
processing, according to parameters, by a guidance neural network model, the task-specific data for a key-frame and the task-specific data for the frame to produce guidance data for a task, wherein the guidance data comprises a global transformation matrix that is regularized as orthogonal; and
applying the guidance data to property data for the key-frame to generate property data for the frame.
7. A system, comprising a processor configured to:
implement a guidance neural network model that is configured to:
receive task-specific data for a key-frame of a video sequence defining attributes of pixels in the key-frame;
receive task-specific data for a frame of the video sequence defining attributes of pixels in the frame; and
process, according to parameters, the task-specific data for a key-frame and the task-specific data for the frame to produce guidance data for a task, wherein the guidance data comprises a global transformation matrix that is regularized as orthogonal; and
apply the guidance data to property data for the key-frame to generate property data for the frame.
8. A computer-implemented method, comprising:
receiving task-specific data for a key-frame of a video sequence defining attributes of pixels in the key-frame;
receiving task-specific data for a frame of the video sequence defining attributes of pixels in the frame;
processing, according to parameters, by a guidance neural network model, the task-specific data for a key-frame and the task-specific data for the frame to produce guidance data for a task; and
applying the guidance data to property data for the key-frame to generate property data for the frame, wherein the task is conversion to high dynamic range, the attributes are low dynamic range data, the property data for the key-frame is high dynamic range data corresponding to the key-frame and the property data for the frame comprises high dynamic range data corresponding to the frame.
9. The computer-implemented method of claim 8 , wherein the guidance data preserves a style energy of the property data for the key-frame in the generated property data for the frame.
10. A system, comprising:
a processor configured to:
implement a guidance neural network model that is configured to:
receive task-specific data for a key-frame of a video sequence defining attributes of pixels in the key-frame;
receive property data for the pixels in the key-frame;
receive task-specific data for a frame of the video sequence defining attributes of pixels in the frame; and
process, according to parameters, the task-specific data for a key-frame and the task-specific data for the frame to produce guidance data for a task, wherein the guidance data comprises at least two affinity values that are aligned in either the horizontal or vertical pixel dimension of the key-frame for transitions from the key-frame to the frame; and
apply the guidance data to the property data for the pixels in the key-frame to generate property data for the pixels in the frame, wherein the guidance data preserves a style energy of the property data for the key-frame in the generated property data for the frame.
11. The computer-implemented method of claim 6 , wherein the guidance data preserves a style energy of the property data for the key-frame in the generated property data for the frame.
12. The system of claim 10 , wherein the guidance data further comprises task-specific affinity values for transitions from the key-frame to other frames in the video sequence.
13. The system of claim 10 , wherein the guidance neural network model is trained during generation of the property data for the frame using a training dataset for the task.
14. The system of claim 10 , wherein the task is colorization, the attributes are lightness data, the property data for the key-frame is color data corresponding to the key-frame and the property data for the frame comprises color data corresponding to the frame.
15. The system of claim 10 , wherein the task is segmentation, the attributes are color data, the property data for the key-frame is segmentation data corresponding to the key-frame and the property data for the frame comprises segmentation data corresponding to the frame.
16. A non-transitory computer-readable media storing computer instructions for spatial linear propagation that, when executed by one or more processors, cause the one or more processors to perform the steps of:
receiving task-specific data for a key-frame of a video sequence defining attributes of pixels in the key-frame;
receiving property data for the pixels in the key-frame;
receiving task-specific data for a frame of the video sequence defining attributes of pixels in the frame;
processing, according to parameters the task-specific data for a key-frame and the task-specific data for the frame to produce guidance data for a task, wherein the guidance data comprises at least two affinity values that are aligned in either the horizontal or vertical pixel dimension of the key-frame for transitions from the key-frame to the frame; and
applying the guidance data to the property data for the pixels in the key-frame to generate property data for the pixels in the frame, wherein the guidance data preserves a style energy of the property data for the key-frame in the generated property data for the frame.
17. The non-transitory computer-readable media of claim 16 , wherein the guidance data preserves a style energy of the property data for the key-frame in the generated property data for the frame.
18. The non-transitory computer-readable media of claim 16 , wherein the guidance data further comprises task-specific affinity values for transitions from the key-frame to other frames in the video sequence.
19. A system, comprising:
a processor configured to:
implement a guidance neural network model that is configured to:
receive task-specific data for a key-frame of a video sequence defining attributes of pixels in the key-frame;
receive task-specific data for a frame of the video sequence defining attributes of pixels in the frame; and
process, according to parameters, the task-specific data for a key-frame and the task-specific data for the frame to produce guidance data for a task; and
apply the guidance data to property data for the key-frame to generate property data for the frame, wherein the task is conversion to high dynamic range, the attributes are low dynamic range data, the property data for the key-frame is high dynamic range data corresponding to the key-frame and the property data for the frame comprises high dynamic range data corresponding to the frame.
20. A non-transitory computer-readable media storing computer instructions for spatial linear propagation that, when executed by one or more processors, cause the one or more processors to perform the steps of:
receiving task-specific data for a key-frame of a video sequence defining attributes of pixels in the key-frame;
receiving task-specific data for a frame of the video sequence defining attributes of pixels in the frame;
processing, according to parameters the task-specific data for a key-frame and the task-specific data for the frame to produce guidance data for a task, wherein the guidance data comprises a global transformation matrix that is regularized as orthogonal; and
applying the guidance data to property data for the key-frame to generate property data for the frame.