Method and device for selecting transform matrices for down-sampling DCT image using learning with forgetting algorithm
View Patent ↗Down-sampling of an image may be performed in the DCT domain. A multiple layered network is used to select transform matrices for down-sampling a DCT image of size M×N to a DCT image of size I×J. A spatial domain down-sampling method is selected and applied to the DCT image to produce a down-sampled DCT reference image. A learning with forgetting algorithm is used to apply a decay to the elements of the transform matrix and select a transform matrices which solve an optimization problem. The optimization problem is a function of the visual quality of images obtained using the transform matrices and the computational complexity associated with using the transform matrices. The visual quality is a measure of the difference between the down-sampled DCT image obtained using the transform matrices and the visual quality of the DCT reference image obtained using a spatial domain down-sampling method.
1. A method of selecting transform matrices for use in a transcoder for down-sampling a DCT image directly in the DCT domain, the method comprising:
obtaining a training set by applying a pre-selected down-sampling method to a DCT reference image to obtain a down-sampled DCT reference image;
initializing the transform matrices; and
applying a learning with forgetting algorithm to the transform matrices to obtain transform matrices which reduce a learning objective function,
wherein the learning objective function is a function of the visual quality of a down-sampled DCT image obtained using the transform matrices relative to the visual quality of the down-sampled DCT reference image, and the learning objective function is also a function of the computational complexity of applying the transform matrices.
2. The method of claim 1 , wherein the step of applying the learning with forgetting algorithm comprises:
computing a learning amount for the transform matrices; and
adjusting the transform matrices using the learning amount.
3. The method of claim 2 , wherein the steps of computing a learning amount and adjusting the transform matrices are repeated until a change in the learning objective function is less than a predetermined amount.
4. The method of claim 1 , wherein the step of applying a learning with forgetting algorithm includes:
passing an input DCT reference image from a training set forward to compute network outputs;
computing the network error;
propagating the network error backwards;
computing learning amounts for the transform matrices; and
adjusting the transform matrices using the learning amounts.
5. The method of claim 1 , wherein the transform matrices comprise a row transform matrix for down-sampling the number of rows in the DCT image and a column size matrix for down-sampling the number of columns in the DCT image.
6. The method of claim 5 , wherein the learning objective function is
J f =∥D IM ·C MN ·W NJ −V IJ ∥ 2 +λ·r g ,
where J f is the learning objective function; D IM is the row transform matrix; W NJ is the column transform matrix; C MN is the DCT image; D IM ·C MN ·W NJ is the down-sampled DCT image obtained using the transform matrices; V IJ is a down-sampled DCT reference image obtained using another down-sampling method; r g is the computational complexity associated with using the transform matrices D IM ,W NJ to down-sample an image C MN ; and λ is a quality and complexity trade-off parameter used to balance a trade-off between visual quality of down-sampled DCT images and the computational complexity of obtaining the down-sampled DCT images.
7. The method of claim 6 , wherein the computational complexity is determined from the absolute values of all elements in the transform matrices.
8. The method of claim 1 further comprising:
applying a selective learning with forgetting algorithm to the transform matrices, the selective learning with forgetting algorithm selectively applying a decay to elements of the transform matrices having weights less than a predetermined selective learning with forgetting threshold parameter and further minimize the learning objective function.
9. The method of claim 8 , wherein the selective learning with forgetting algorithm is repeatedly applied to the transform matrices until the change in a learning objective function is less than a second predetermined threshold.
10. The method of claim 9 , wherein the learning objective function used in the selective learning with forgetting algorithm is
J f =∥D IM ·C MN ·W NJ −V IJ ∥ 2 +λ·r q ,
where J f is the learning objective function; D IM is the row transform matrix; W NJ is the column transform matrix; C MN is the DCT image; D IM ·C MN ·W NJ is the down-sampled DCT image obtained using the transform matrices; V IJ is the down-sampled DCT reference image obtained by down-sampling the DCT image C MN using another down-sampling method; r q is a measure of the selective learning with forgetting computational complexity associated with using the transform matrices D IM ,W NJ to down-sample an image C MN ; and λ is a quality and complexity trade-off parameter which is used to obtain a desired trade-off between visual quality of the down-sampled DCT image and the computational complexity of obtaining the down-sampled DCT image.
11. The method of claim 10 , wherein the selective learning with forgetting computational complexity is a function of the absolute values of all elements in the row transform matrix and the column transform matrix that are below the selective learning with forgetting threshold parameter, and wherein the step of training further comprises:
determining the selective learning with forgetting computational complexity of the transform matrices.
12. The method of claim 11 , wherein the selective learning with forgetting computational complexity is also a function of the number of non-zero elements in a binary representation of elements of the transform matrices.
13. An electronic device comprising:
a memory for storing transform matrices and a training set, the transform matrices comprising a row transform matrix and a column transform matrix, the training set comprising at least one DCT reference image and a corresponding down-sampled DCT reference image obtained using a spatial domain down-sampling method; and
a processor connected to the memory for obtaining transform matrices for down-sampling a DCT image directly in a DCT domain, the processor being configured to initialize the transform matrices and to apply a learning with forgetting algorithm to the transform matrices to obtain transform matrices which reduce a learning objective function, the learning objective function being a function of the visual quality of a down-sampled DCT image obtained using the transform matrices relative to the visual quality of the down-sampled DCT reference image, and the learning objective function also being a function of the computational complexity associated with down-sampling using the transform matrices.
14. The electronic device of claim 13 , wherein the processor is further configured to compute a learning amount for the transform matrices and adjust the transform matrices using the learning amount.
15. The electronic device of claim 14 , wherein the processor is further configured to repeatedly compute a learning amount and adjust the transform matrices until a change in the learning objective function is less than a predetermined amount.
16. The electronic device of claim 13 , wherein the learning objective function is
J f =∥D IM ·C MN ·W NJ −V IJ ∥ 2 +λ·r g ,
where J f is the learning objective function; D IM is the row transform matrix; W NJ is the column transform matrix; C MN is the DCT image; D IM ·C MN ·W NJ is the down-sampled DCT image obtained using the transform matrices; V IJ is a down-sampled DCT reference image obtained using another down-sampling method; r g is the computational complexity associated with using the transform matrices D IM ,W NJ to down-sample an image C MN ; and λ is a quality and complexity trade-off parameter used to balance a trade-off between visual quality of down-sampled DCT images and the computational complexity of obtaining the down-sampled DCT images.
17. The electronic device of claim 13 wherein the processor is further configured to apply a selective learning with forgetting algorithm to the transform matrices, the selective learning with forgetting algorithm selectively applying a decay to elements of the transform matrices having weights less than a predetermined selective learning with forgetting threshold parameter and further minimize the learning objective function.
18. The electronic device of claim 13 , wherein the processor is configured to determine transform matrices when there is low processor usage.
19. The electronic device of claim 13 , wherein the electronic device is a server which may be accessed by one or more wireless devices.
20. The electronic device of claim 13 , wherein the DCT image is a frame of a DCT video.