Lightweight dense residual network for video super-resolution on mobile devices
The technology described herein relates to a lightweight dense residual network for video super-resolution on mobile devices. A method for implementing a lightweight dense residual network to achieve super-resolution performance may include generating feature maps using a network based on an input of frames at a lower resolution, the network comprised of DenseRes blocks and an additional convolution operation, each DenseRes block comprising multiple layers of convolution operations and rectified linear activation function (ReLU) operations and a 1×1 convolution operation. Said feature maps are upsampled by a pixel shuffle layer in the network and the frames are output at a higher resolution, the higher resolution relative to the lower resolution by an upscaling factor.
1 . A method for implementing a lightweight dense residual network to achieve super-resolution performance, comprising:
receiving a video input comprising a number of frames at a lower resolution;
generating a first output in real-time on a mobile device, by a network comprising two or more DenseRes blocks and a 1×1 convolution operation, each DenseRes block comprising two or more layers, the two or more layers comprising a convolution operation, a rectified linear activation function (ReLU) operation, and an additional convolution operation, each DenseRes block configured to add a feature map from a previous layer to an upcoming layer without concatenation, the first output comprising a plurality of feature maps;
upsampling the first output by a pixel shuffle layer in the network; and
generating a second output comprising the number of frames at a higher resolution, the higher resolution relative to the lower resolution by an upscaling factor.
2 . The method of claim 1 , wherein the network comprises an additional ReLU operation clipped to have a maximum value of 1, the additional ReLU operation implemented before the pixel shuffle layer.
3 . The method of claim 1 , wherein the network is configured to run in real-time on a mobile device.
4 . The method of claim 1 , wherein the second output provides for video super-resolution on a mobile device.
5 . The method of claim 1 , wherein the network is configured to process ten or more frames concurrently.
6 . The method of claim 5 , wherein the ten or more frames comprises every sixth frame in 60 FPS video.
7 . The method of claim 1 , wherein a residual connection from the previous layer propagates a feature map from the previous layer to one or more upcoming layers.
8 . The method of claim 1 , further comprising extracting a compressed feature map, by the 1×1 convolution operation, from two or more feature maps within a DenseRes block.
9 . A non-transitory computer-readable medium storing computer instructions for scalable per-title encoding that when executed on one or more computer processors perform the steps of:
receiving a video input comprising a number of frames at a lower resolution;
generating a first output, by a network comprising two or more DenseRes blocks and a 1×1 convolution operation, each DenseRes block comprising two or more layers, each of the two or more layers comprising a convolution operation, a rectified linear activation function (ReLU) operation, and an additional convolution operation, each DenseRes block configured to add a feature map from a previous layer to an upcoming layer without concatenation, the first output comprising a plurality of feature maps;
upsampling the first output by a pixel shuffle layer in the network; and
generating a second output comprising the number of frames at a higher resolution, the higher resolution relative to the lower resolution by an upscaling factor.