Artificial intelligence image frame processing systems and methods
Systems and method for streaming video content include downscaling video content using a downscaling model to generate downscaled video content and downloading the downscaled video content as a video stream and corresponding upscaling model to a client device. The system converts received video frames to a video memory format comprising channels having the same memory allocation size, each subsequent channel arranged in an adjacent memory location, for input to the downscaling model. The client device upscales the video stream using the received upscaling model for display by the client device in real-time. A training system trains the downscaling model to generate the downscaled video content, based on associated metadata identifying a type of video content. The downscaled video content and associated upscaling models are stored for access by an edge server, which downloads upscaling models to a client device to select an upscaling model.
1. A method for streaming video content comprising:
converting received video frames to a video memory format comprising a plurality of channels representing a color space, wherein each of the plurality of channels has the same memory allocation size and is associated with a respective luma component or chroma component of the color space, wherein each subsequent channel of the plurality of channels is arranged in an adjacent memory location, and wherein the video memory format for an N×M video frame of the received video frames comprises 3×N columns and M/2 rows;
downscaling the video frames stored in the video memory format using a downscaling neural network model to generate downscaled video content; and
transmitting the downscaled video content as a video stream to a client device.
2. The method of claim 1 , further comprising training the downscaling neural network model to generate the downscaled video content.
3. The method of claim 1 , wherein the downscaled video content includes associated metadata identifying a type of video content, and wherein the downscaling neural network model is trained to generate the downscaled video content for the type of video content.
4. The method of claim 1 , wherein the downscaled video content and one or more upscaling models associated with the downscaled video content are stored for access by an edge server, and wherein transmitting the downscaled video content as the video stream and the one or more upscaling models associated with the downscaled video content is performed by the edge server.
5. The method of claim 4 , wherein the edge server transmits a plurality of upscaling models to the client device, and wherein the client device is configured to select an upscaling model for use by the client device.
6. The method of claim 1 , wherein the method is performed by a video streaming system.
7. The method of claim 1 , further comprising initiating a video conferencing session.
8. A system comprising:
a processor; and
a memory storing instructions that, when executed by the processor, cause the system to:
convert received video frames to a video memory format comprising a plurality of channels representing a color space, wherein each of the plurality of channels has the same memory allocation size and is associated with a respective luma component or chroma component of the color space, wherein each subsequent channel of the plurality of channels is arranged in an adjacent memory location, and wherein the video memory format for an N×M video frame of the received video frames comprises 3×N columns and M/2 rows;
downscale the video frames stored in the video memory format using a downscaling neural network model to generate downscaled video content; and
transmit the downscaled video content as a video stream to a client device.
9. The system of claim 8 , wherein execution of the instructions further causes the system to:
transmit the downscaled video content and an upscaling model associated with the downscaled video content to an edge server.
10. The system of claim 9 , wherein execution of the instructions further causes the system to generate the upscaling model.
11. The system of claim 10 , wherein execution of the instructions further causes the system to:
detect a video content type and train the upscaling model to optimize upscaling of video for the video content type.
12. The system of claim 10 , wherein execution of the instructions further causes the system to:
train a downscale model to receive video content; and
generate the downscaled video content for streaming.
13. The system of claim 8 , wherein the video content includes associated metadata identifying a type of video content, and wherein the downscaling neural network model is trained to generate the downscaled video content for the type of video content.
14. The system of claim 8 , wherein an edge server is configured to transmit a plurality of upscaling models to the client device, and wherein the client device is configured to select an upscaling model for use by the client device in preparing the video stream for display.
15. The system of claim 8 , wherein the system is a video streaming system.
16. The system of claim 8 , wherein the system is a video conferencing session.