Visual processing using sub-pixel convolutions
A method for enhancing one or more sections of lower-quality visual data using a hierarchical algorithm, the method comprising receiving one or more sections of lower-quality visual data. The one or more sections of lower-quality visual data are enhanced to one or more sections of higher-quality visual data using the hierarchical algorithm. Additionally, at least the first step of the hierarchical algorithm is performed in a lower-quality domain; and wherein the hierarchical algorithm operates in both a higher-quality domain and the lower-quality domain.
1. A method for enhancing one or more sections of lower-resolution visual data using a convolutional neural network, the method comprising the steps of:
receiving one or more sections of lower-resolution visual data; and
enhancing the one or more sections of lower-resolution visual data to one or more sections of higher-resolution visual data using the convolutional neural network, the convolutional neural network comprising a plurality of connected layers followed by a sub-pixel convolution layer,
wherein at least an initial layer of the convolutional neural network is performed in a lower-resolution domain, and
wherein a later layer of the convolutional neural network operates in a higher-resolution domain.
2. The method of claim 1 , wherein the initial layer of the convolutional neural network comprises extracting one or more features from the one or more sections of lower-resolution visual data.
3. The method of claim 1 , wherein the later layer is a last layer of the convolutional neural network, which comprises the sub-pixel convolution layer that produces higher-resolution visual data corresponding to the lower-resolution visual data.
4. The method of claim 3 , wherein the last layer of the convolutional neural network includes periodic shuffling.
5. The method of claim 1 , wherein the plurality of connected layers are non-linear mapping layers.
6. The method of claim 1 , wherein the plurality of connected layers is selected from a group that includes: sequential, recurrent, recursive, branching, and merging.
7. The method of claim 1 , wherein a section of visual data is selected from a group that includes: a single frame of visual data, a sequence of frames of visual data, and a region within a sequence of frames of visual data.
8. The method of claim 1 , wherein the lower-resolution visual data comprises a plurality of frames of video, or a plurality of images.
9. The method of claim 1 , wherein the convolutional neural network differs for each section of visual data.
10. The method of claim 1 , wherein standardised features of the one or more sections of received lower-resolution visual data are extracted and used to select the convolutional neural network from a library of networks.
11. The method of claim 1 , wherein the convolutional neural network is selected from a library of networks as the convolutional neural network generating a highest quality version of the higher-resolution visual data, and wherein quality is defined by a group including: an error rate; a bit error rate; a peak signal-to-noise ratio; and a structural similarity index.
12. The method of claim 1 , wherein the convolutional neural network is developed using a learned approach.
13. The method of claim 1 , wherein a stride of the initial layer is 1/r, where r is an upscaling factor.
14. The method of claim 1 , wherein the convolutional neural network comprises two or more convolutional neural networks.
15. The method of claim 1 , wherein the convolutional neural network can be used as a filter in encoding or decoding of visual data.
16. The method of claim 1 , wherein the convolutional neural network reduces artefacts in the higher-resolution visual data.
17. The method of claim 1 , wherein the convolutional neural network uses a spatio-temporal approach.
18. A computer program product embodied on a non-transitory storage medium and comprising instructions that, when executed, cause a system to enhance at least a section of lower-resolution visual data using a convolutional neural network, by performing the steps of:
receiving one or more sections of lower-resolution visual data; and
enhancing the one or more sections of lower-resolution visual data to one or more sections of higher-resolution visual data using the convolutional neural network, the convolutional neural network comprising a plurality of connected layers followed by a sub-pixel convolution layer,
wherein a first layer of the convolutional neural network includes extracting one or more features from the one or more sections of lower-resolution visual data in a lower-resolution domain, and
wherein a later layer of the convolutional neural network operates in a higher-resolution domain.
19. The computer program product of claim 18 , wherein the plurality of connected layers are non-linear mapping layers.
20. The computer program product of claim 18 , wherein the later layer is a last layer of the convolutional neural network.