Real-time temporally consistent object segmented style transfer in media and gaming
One embodiment provides a method comprising, at a runtime library executed by a processor of a data processing system, receiving an input frame having objects to be stylized via a style transfer network associated with the runtime library, wherein the style transfer network is a neural network model trained to apply one or more visual styles to an input frame, performing instance segmentation on the input frame to generate one or more instance masks to identify one or more objects to be stylized, generating one or more stylized frames for each style to transfer to the input frame, and merging, via the one or more instance masks, stylized objects from one or more stylized frames with un-stylized content from the input frame to generate an output frame with per-instance stylization.
1. A method comprising:
at a runtime library executed by a processor of a data processing system:
receiving an input frame having objects to be stylized via a style transfer network associated with the runtime library, wherein the style transfer network is a neural network model trained to apply one or more visual styles to an input frame;
performing instance segmentation on the input frame to generate multiple instance masks to identify multiple objects to be stylized;
generating multiple stylized frames for each style to transfer to the input frame, wherein each of the multiple stylized frames is stylized to a separate style; and
merging, via the multiple instance masks, stylized objects from the multiple stylized frames with un-stylized content from the input frame to generate an output frame with per-instance stylization of multiple objects.
2. The method as in claim 1 , wherein one or more identifiers of objects to be stylized are received in conjunction with the input frame.
3. The method as in claim 1 , wherein the input frame is a video frame of video content.
4. The method as in claim 3 , wherein the video content is streaming media content.
5. The method as in claim 1 , wherein the input frame is a frame of a three-dimensional graphics application rendered by a graphics processor.
6. The method as in claim 5 , wherein the data processing system is a cloud gaming server configured to stylize cloud gaming output via the runtime library.
7. The method as in claim 5 , further comprising performing instance segmentation on the input frame based on object data provided by a render associated with the three-dimensional graphics application.
8. The method as in claim 5 , further comprising performing instance segmentation on the input frame based on object data provided by a shim layer for an application programming interface (API) used to render the three-dimensional graphics application.
9. The method as in claim 8 , further comprising:
intercepting a call to the API by the three-dimensional graphics application via the shim layer; and
adding an additional render pass to capture the object data.
10. A non-transitory machine-readable medium storing instructions which, when executed by one or more processors of a data processing system, cause the one or more processors to provide a runtime library, wherein the instructions cause the one or more processors to perform operations comprising:
receiving an input frame having objects to be stylized via a style transfer network associated with the runtime library, wherein the style transfer network is a neural network model trained to apply one or more visual styles to an input frame;
performing instance segmentation on the input frame to generate multiple instance masks to identify multiple objects to be stylized;
generating multiple stylized frames for each style to transfer to the input frame, wherein each of the multiple stylized frames is stylized to a separate style; and
merging, via the multiple instance masks, stylized objects from the multiple stylized frames with un-stylized content from the input frame to generate an output frame with per-instance stylization of multiple objects.
11. The non-transitory machine-readable medium as in claim 10 , wherein one or more identifiers of objects to be stylized are received in conjunction with the input frame.
12. The non-transitory machine-readable medium as in claim 10 , wherein the input frame is a video frame of video content.
13. The non-transitory machine-readable medium as in claim 12 , wherein the video content is streaming media content.
14. The non-transitory machine-readable medium as in claim 10 , wherein the input frame is a frame of a three-dimensional graphics application rendered by a graphics processor.
15. The non-transitory machine-readable medium as in claim 14 , wherein the data processing system is a cloud gaming server configured to stylize cloud gaming output via the runtime library.
16. The non-transitory machine-readable medium as in claim 14 , the operations further comprising performing instance segmentation on the input frame based on object data provided by a render associated with the three-dimensional graphics application.
17. The non-transitory machine-readable medium as in claim 14 , the operations further comprising performing instance segmentation on the input frame based on object data provided by a shim layer for an application programming interface (API) used to render the three-dimensional graphics application.
18. The non-transitory machine-readable medium as in claim 17 , the operations further comprising:
intercepting a call to the API by the three-dimensional graphics application via the shim layer; and
adding an additional render pass to capture the object data.
19. A data processing system comprising:
a memory device; and
one or more processors to execute instructions stored on the memory device, the one or more processors including a graphics processor, wherein the instructions cause the one or more processors to perform operations comprising:
receiving an input frame having objects to be stylized via a style transfer network associated with a runtime library provided by the instructions stored on the memory device, wherein the style transfer network is a neural network model trained to apply one or more visual styles to an input frame;
performing instance segmentation on the input frame to generate multiple instance masks to identify multiple objects to be stylized;
generating multiple stylized frames for each style to transfer to the input frame, wherein each of the multiple stylized frames is stylized to a separate style; and
merging, via the multiple instance masks, stylized objects from the multiple stylized frames with un-stylized content from the input frame to generate an output frame with per-instance stylization of multiple objects.
20. The data processing system as in claim 19 , wherein the input frame is a video frame of video content or a frame of a three-dimensional graphics application rendered by the graphics processor.