Video bandwidth optimization within a video communications platform
Methods, systems, and apparatus, including computer programs encoded on computer storage media related to video bandwidth optimization, and more particularly, to systems and methods for video resolution downscaling and upscaling a video communications platform. A first video stream comprising first multiple image frames of a first resolution is received. A modified video stream of a second resolution higher than the first resolution, is generated using a trained machine learning network. A user interface may display the modified video stream.
1. A computer-implemented method comprising:
transmitting, by a first client device, a resolution parameter to a second client device, the resolution parameter indicating a first resolution;
receiving, by the first client device, one or more downscaled images from the second client device, the one or more downscaled images being at the first resolution;
determining which upscaling image process to use to upscale the received one or more downscaled images based on a selected image resolution output, wherein the determination of the upscaling image process is between a machine learning model that is trained using image pairs depicting a same image at different resolutions and a non-machine learning model based process;
selecting the upscaling image process that uses the machine learning model and generating a first set of upscaled images by:
inputting the received one or more downscaled images into the machine learning model; and
upscaling the received one or more downscaled images by the machine learning model, wherein upscaling comprises:
generating a first input sub-image of a respective input image, the first input sub-image corresponding to brightness information of the respective input image; and
generating a second input sub-image of the respective input image, the second input sub-image corresponding to color information of the respective input image; and
outputting by the machine learning model, one or more images based on the selected image resolution output, the one or more images being at a second resolution that is at a resolution higher than the first resolution; and
displaying, via a user interface of the first client device, the generated first set of upscaled images.
2. The computer-implemented method of claim 1 , wherein the first resolution is any one of a resolution of 180p, 270p, 360p, 480p and 720p, and the second resolution is 1080p.
3. The computer-implemented method of claim 1 , further comprising:
receiving additional one or more images from the second client device, the additional one or more images being at the same resolution as the first resolution or a different resolution than the first resolution;
generating a second set of upscaled images by:
inputting the additional received one or more images into the machine learning model;
upscaling the additional received one or more image frames by the machine learning model; and
outputting by the machine learning model, additional one or more images at the second resolution; and
displaying, via the user interface of the first client device, the second set of upscaled images frames.
4. The computer-implemented method of claim 1 , further comprising:
receiving a video stream from a camera operable with the first client device, the video stream including multiple images at a resolution of 1080p or 720p;
downscaling the multiple images from the resolution of 1080p or 720p to a lower resolution of 180p, 270p, 360p or 480p; and
transmitting, from the first client device, the downscaled multiple images at the lower resolution to the second client device.
5. The computer-implemented method of claim 1 , further comprising:
training the machine learning model with multiple image pairs wherein the image pairs include an image at the first resolution and a corresponding image at the second resolution, and wherein the machine learning model is trained to receive an input of an image at the first resolution and output an image at the second resolution, the second resolution being a higher resolution than the first resolution.
6. The computer-implemented method of claim 1 , wherein the machine learning model comprises a convolutional neural network (CNN) including:
a multiple channel input to receive an input image into a series of feature extraction blocks; and
multiple upsample blocks, wherein a final upsample block outputs an image of a resolution higher than the input image, wherein each of the feature extraction blocks include a first convolution block to increase a number of channels of the CNN and a second convolutional block to decrease the number of channels of the CNN.
7. The computer-implemented method of claim 1 , wherein inputting the received one or more images into the trained machine learning model comprises:
decompressing a particular image included in the received one or more downscaled images;
identifying one or more pixel changing areas and one or more pixel non-changing areas in a decompressed version of the particular image; and
discarding each respective pixel non-changing area in the decompressed version of the particular image from a machine learning model input sourced from the particular image; and
selecting the one or more pixel changing areas as the machine learning model input sourced from the particular image.
8. The computer-implemented method of claim 1 , wherein outputting by the trained machine learning model, one or more images at a second resolution comprises:
applying one or more feature extraction blocks and one or more upsample blocks to the first input sub-image;
applying a bilinear upsampling process to the second input sub-image;
generating a composite upsampled image based at least in part on the first and the second input sub-images; and
outputting the composite upsampled image.
9. A non-transitory computer readable medium that stores executable program instructions that when executed by one or more computing devices configure the one or more computing devices to perform operations comprising:
transmitting, by a first client device, a resolution parameter to a second client device, the resolution parameter indicating a first resolution;
receiving, by the first client device, one or more downscaled images from the first client device, the one or more downscaled images being at the first resolution;
determining which upscaling image process to use to upscale the received one or more downscaled images based on a selected image resolution output, wherein the determination of the upscaling image process is between a machine learning model that is trained using image pairs depicting a same image at different resolutions and a non-machine learning model based process;
selecting the upscaling image process that uses the machine learning model and generating a first set of upscaled images by:
inputting the received one or more downscaled images into the machine learning model; and
upscaling the received one or more downscaled images by the machine learning model, wherein upscaling comprises:
generating a first input sub-image of a respective input image, the first input sub-image corresponding to brightness information of the respective input image; and
generating a second input sub-image of the respective input image, the second input sub-image corresponding to color information of the respective input image; and
outputting by the machine learning model, one or more images based on the selected image resolution output, the one or more images being at a second resolution that is at a resolution higher than the first resolution; and
displaying, via a user interface of the first client device, the generated first set of upscaled images.
10. The non-transitory computer readable medium of claim 9 , wherein the first resolution is any one of a resolution of 180p, 270p, 360p, 480p and 720p, and the second resolution is 1080p.
11. The non-transitory computer readable medium of claim 9 , further comprising:
receiving additional one or more images from the second client device, the additional one or more images being at the same resolution as the first resolution or a different resolution than the first resolution;
generating a second set of upscaled images by:
inputting the additional received one or more images into the machine learning model;
upscaling the additional received one or more image frames by the machine learning model; and
outputting by the trained machine learning model, additional one or more images at the second resolution; and
displaying, via the user interface of the first client device, the second set of upscaled images frames.
12. The non-transitory computer readable medium of claim 9 , further comprising:
receiving a video stream from a camera operable with the first client device, the video stream including multiple images at a resolution of 1080p or 720p;
downscaling the multiple images from the resolution of 1080p or 720p to a lower resolution of 180p, 270p, 360p or 480p; and
transmitting, from the first client device, the downscaled multiple images at the lower resolution to the second client device.
13. The non-transitory computer readable medium of claim 9 , further comprising:
training the machine learning model with multiple image pairs wherein the image pairs include an image at the first resolution and a corresponding image at the second resolution, and wherein the machine learning model is trained to receive an input of an image at the first resolution and output an image at the second resolution, the second resolution being a higher resolution than the first resolution.
14. The non-transitory computer readable medium of claim 9 , wherein the machine learning model comprises a convolutional neural network (CNN) including:
a multiple channel input to receive an input image into a series of feature extraction blocks; and
multiple upsample blocks, wherein a final upsample block outputs an image of a resolution higher than the input image.
15. The non-transitory computer readable medium of claim 14 , wherein each of the feature extraction blocks include a first convolution block to increase a number of channels of the CNN and a second convolutional block to decrease the number of channels of the CNN.
16. A system comprising one or more processors configured to perform the operations of:
transmitting, by a first client device, a resolution parameter to a second client device, the resolution parameter indicating a first resolution;
receiving, by the first client device, one or more downscaled images from the second client device, the one or more downscaled images being at the first resolution;
determining which upscaling image process to use to upscale the received one or more downscaled images based on a selected image resolution output, wherein the determination of the upscaling image process is between a machine learning model that is trained using image pairs depicting a same image at different resolutions and a non-machine learning model based process;
selecting the upscaling image process that uses the machine learning model and generating a first set of upscaled images by:
inputting the received one or more downscaled images into the machine learning model; and
upscaling the received one or more downscaled images by the machine learning model, wherein upscaling comprises:
generating a first input sub-image of a respective input image, the first input sub-image corresponding to brightness information of the respective input image; and
generating a second input sub-image of the respective input image, the second input sub-image corresponding to color information of the respective input image; and
outputting by the machine learning model, one or more images based on the selected image resolution output, the one or more images being at a second resolution that is at a resolution higher than the first resolution; and
displaying, via a user interface of the first client device, the generated first set of upscaled images.
17. The system of claim 16 , the operations further comprising:
receiving additional one or more images from the second client device, the additional one or more images being at the same resolution as the first resolution or a different resolution than the first resolution;
generating a second set of upscaled images by:
inputting the additional received one or more images into the machine learning model;
upscaling the additional received one or more image frames by the machine learning model; and
outputting by the machine learning model, additional one or more images at the second resolution; and
displaying, via the user interface of the first client device, the second set of upscaled images frames.
18. The system of claim 16 , the operations further comprising:
wherein the first resolution is any one of a resolution of 180p, 270p, 360p, 480p and 720p, and the second resolution is 1080p;
receiving a video stream from a camera operable with the first client device, the video stream including multiple images at a resolution of 1080p or 720p;
downscaling the multiple images from the resolution of 1080p or 720p to a lower resolution of 180p, 270p, 360p or 480p; and
transmitting, from the first client device, the downscaled multiple images at the lower resolution to the second client device.
19. The system of claim 16 , the operations further comprising:
training the machine learning model with multiple image pairs, wherein the image pairs include an image at the first resolution and a corresponding image at the second resolution, and wherein the machine learning model is trained to receive an input of an image at the first resolution and output an image at the second resolution, the second resolution being a higher resolution than the first resolution.
20. The system of claim 16 , wherein the machine learning model comprises a convolutional neural network (CNN) including:
a multiple channel input to receive an input image into a series of feature extraction blocks; and
multiple upsample blocks, wherein a final upsample block outputs an image of a resolution higher than the input image.