Granted Patent B2

US 12,236,556 · App. 17/762,199 · Granted Feb 25, 2025

Inventors: Lijie Zhang (Beijing, CN); Dan Zhu (Beijing, CN)

Assignee: BOE TECHNOLOGY GROUP CO., LTD.

G06T3/4053G06T5/20G06V10/82

Abstract

Claims (91)

1. A method for enhancing a video resolution, comprising:

obtaining multiple frames of images as input data, and obtaining initial data by performing feature extraction on the input data using a first three-dimensional convolutional layer;

obtaining first feature data by performing down-sampling on the initial data at a preset multiple;

obtaining first reference data by performing a convolution operation on the first feature data using a second three-dimensional convolutional layer to merge the first feature data into one frame; and

obtaining first output data by performing up-sampling on the first reference data at the preset multiple;

wherein the method further comprises:

performing an Nth super-resolution operation on the first feature data, the super-resolution operation comprising a down-sampling operation, a first feature extraction operation, a merging operation, a second feature extraction operation, and an up-sampling operation, wherein

the down-sampling operation comprises performing down-sampling on the first feature data at the preset multiple;

the first feature extraction operation comprises performing the first feature extraction operation on the down-sampled first feature data by using the first three-dimensional convolution layer to obtain third feature data;

the merging operation comprises performing a convolution operation on the third feature data by using the second three-dimensional convolutional layer to merge the third feature data into one frame to obtain second reference data;

the second feature extraction operation comprises performing the second feature extraction operation on stacked data of the second reference data and (N+1)th output result by using the first three-dimensional convolution layer to obtain fourth feature data; and

the up-sampling operation comprises performing up-sampling on the fourth feature data at the preset multiple to obtain third output data; and

updating the first reference data with the third output data;

wherein an input of Nth down-sampling operation is an output of the first feature extraction operation of (N−1)th super-resolution operation, and N is a positive integer starting from 1.

2. The method according to claim 1 , wherein the obtaining the first feature data by performing down-sampling on the initial data at the preset multiple comprises:

performing down-sampling on the initial data at the preset multiple, and

obtaining the first feature data by performing feature extraction on the down-sampled initial data using the first three-dimensional convolutional layer.

3. The method according to claim 1 , further comprising:

obtaining second output data by performing a convolution operation on the initial data using the second three-dimensional convolution layer to merge the initial data into one frame;

obtaining target data by stacking the second output data and the first output data and performing a convolution operation using the first three-dimensional convolution layer; and

obtaining a super-resolution image by performing multiple up-sampling on the target data.

4. The method according to claim 1 , wherein the obtaining the first output data by performing up-sampling on the first reference data at the preset multiple comprises:

obtaining second feature data by performing feature extraction on the first reference data using the first three-dimensional convolution layer; and

obtaining the first output data by performing up-sampling on the second feature data at the preset multiple.

5. The method according to claim 1 , wherein the updating the first reference data with the third output data comprises:

updating the third output data output by (N−1)th up-sampling operation using an output of Nth up-sampling operation; and

obtaining a updated first reference data by stacking the third output data and the first reference data.

6. The method according to claim 1 , wherein at least one characteristic parameter of the input data comprises a quantity of channels, a batch size, and a height, width and time of each frame of the image; and

wherein the batch size is a quantity of the input data simultaneously input.

7. The method according to claim 1 , wherein the preset multiple is an even number.

8. A non-transitory computer-readable storage medium on which a computer program is stored, wherein the program, when executed by a processor, implements a method for enhancing a video resolution comprising:

obtaining multiple frames of images as input data, and obtaining initial data by performing feature extraction on the input data using a first three-dimensional convolutional layer;

obtaining first feature data by performing down-sampling on the initial data at a preset multiple;

obtaining first reference data by performing a convolution operation on the first feature data using a second three-dimensional convolutional layer to merge the first feature data into one frame; and

obtaining first output data by performing up-sampling on the first reference data at the preset multiple;

wherein the method further comprises:

performing an Nth super-resolution operation on the first feature data, the super-resolution operation comprising a down-sampling operation, a first feature extraction operation, a merging operation, a second feature extraction operation, and an up-sampling operation; wherein

the down-sampling operation comprises performing down-sampling on the first feature data at the preset multiple;

the first feature extraction operation comprises performing the first feature extraction operation on the down-sampled first feature data by using the first three-dimensional convolution layer to obtain third feature data;

the merging operation comprises performing a convolution operation on the third feature data by using the second three-dimensional convolutional layer to merge the third feature data into one frame to obtain second reference data;

the second feature extraction operation comprises performing the second feature extraction operation on stacked data of the second reference data and (N+1)th output result by using the first three-dimensional convolution layer to obtain fourth feature data; and

the up-sampling operation comprises performing up-sampling on the fourth feature data at the preset multiple to obtain third output data; and

updating the first reference data with the third output data;

wherein an input of Nth down-sampling operation is an output of the first feature extraction operation of (N−1)th super-resolution operation, and N is a positive integer starting from 1.

9. The non-transitory computer-readable storage medium according to claim 8 , wherein the obtaining the first feature data by performing down-sampling on the initial data at the preset multiple comprises:

performing down-sampling on the initial data at the preset multiple, and

obtaining the first feature data by performing feature extraction on the down-sampled initial data using the first three-dimensional convolutional layer.

10. The non-transitory computer-readable storage medium according to claim 8 , wherein the obtaining the first output data by performing up-sampling on the first reference data at the preset multiple comprises:

obtaining second feature data by performing feature extraction on the first reference data using the first three-dimensional convolution layer; and

obtaining the first output data by performing up-sampling on the second feature data at the preset multiple.

11. The non-transitory computer-readable storage medium according to claim 8 , wherein the updating the first reference data with the third output data comprises:

updating the third output data output by (N−1)th up-sampling operation using an output of Nth up-sampling operation; and

obtaining a updated first reference data by stacking the third output data and the first reference data.

12. The non-transitory computer-readable storage medium according to claim 8 , wherein at least one characteristic parameter of the input data comprises a quantity of channels, a batch size, and a height, width and time of each frame of the image; and

wherein the batch size is a quantity of the input data simultaneously input.

13. An electronic device, comprising:

one or more processors; and

a memory configured to store one or more programs which, when executed by the one or more processors, cause the one or more processors to:

obtain multiple frames of images as input data, and obtain initial data by performing feature extraction on the input data using a first three-dimensional convolutional layer;

obtain first feature data by performing down-sampling on the initial data at a preset multiple;

obtain first reference data by performing a convolution operation on the first feature data using a second three-dimensional convolutional layer to merge the first feature data into one frame; and

obtain first output data by performing up-sampling on the first reference data at the preset multiple;

wherein the one or more processors are further configured to:

perform an Nth super-resolution operation on the first feature data, the super-resolution operation comprising a down-sampling operation, a first feature extraction operation, a merging operation, a second feature extraction operation, and an up-sampling operation; wherein

the down-sampling operation comprises performing down-sampling on the first feature data at the preset multiple;

the first feature extraction operation comprises performing the first feature extraction operation on the down-sampled first feature data by using the first three-dimensional convolution layer to obtain third feature data;

the merging operation comprises performing a convolution operation on the third feature data by using the second three-dimensional convolutional layer to merge the third feature data into one frame to obtain second reference data;

the second feature extraction operation comprises performing the second feature extraction operation on stacked data of the second reference data and (N+1)th output result by using the first three-dimensional convolution layer to obtain fourth feature data; and

the up-sampling operation comprises performing up-sampling on the fourth feature data at the preset multiple to obtain third output data; and

update the first reference data with the third output data;

wherein an input of Nth down-sampling operation is an output of the first feature extraction operation of (N−1)th super-resolution operation, and N is a positive integer starting from 1.

14. The electronic device according to claim 13 , wherein the one or more processors are further configured to:

perform down-sampling on the initial data at the preset multiple, and

obtain the first feature data by performing feature extraction on the down-sampled initial data using the first three-dimensional convolutional layer.

15. The electronic device according to claim 13 , wherein the one or more processors are further configured to:

obtain second output data by performing a convolution operation on the initial data using the second three-dimensional convolution layer to merge the initial data into one frame;

obtain target data by stacking the second output data and the first output data and performing a convolution operation using the first three-dimensional convolution layer; and

obtain a super-resolution image by performing multiple up-sampling on the target data.

16. The electronic device according to claim 13 , wherein the one or more processors are further configured to:

obtain second feature data by performing feature extraction on the first reference data using the first three-dimensional convolution layer; and

obtain the first output data by performing up-sampling on the second feature data at the preset multiple.

17. The electronic device according to claim 13 , wherein the one or more processors are further configured to:

update the third output data output by (N−1)th up-sampling operation using an output of Nth up-sampling operation; and

obtain a updated first reference data by stacking the third output data and the first reference data.

18. The electronic device according to claim 13 , wherein at least one characteristic parameter of the input data comprises a quantity of channels, a batch size, and a height, width and time of each frame of the image; and

wherein the batch size is a quantity of the input data simultaneously input.

19. The electronic device according to claim 13 , wherein the preset multiple is an even number.

20. The non-transitory computer-readable storage medium according to claim 17 , wherein method further comprises:

obtaining second output data by performing a convolution operation on the initial data using the second three-dimensional convolution layer to merge the initial data into one frame;

obtaining target data by stacking the second output data and the first output data and performing a convolution operation using the first three-dimensional convolution layer; and

obtaining a super-resolution image by performing multiple up-sampling on the target data.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (22)

Cited By (2)

US 12,718,322 US 12,739,343