IP Library › Granted Patent US 11,558,631
Granted Patent B2
US 11,558,631 · App. 16/836,078 · Granted Jan 17, 2023

Super-resolution loop restoration

Inventors: Urvang Joshi (Mountain View, CA); Debargha Mukherjee (Cupertino, CA); Andrew Simpson (San Francisco, CA)
Assignee: GOOGLE LLC
H04N19/50H04N7/0117H04N19/117H04N19/176H04N19/184H04N19/619
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,558,631
App. No.
16/836,078
Granted
Jan 17, 2023
Kind
B2
Abstract

A super-resolution coding mode is described. Encoded image can be decoded by decoding, from an encoded bitstream, a flag indicating whether an image was encoded using the super-resolution mode. The image is encoded at a first resolution. Responsive to the flag indicating that the image was encoded using the super-resolution mode, bits indicating an amount of scaling of the image are decoded. The image is decoded from the encoded bitstream to obtain a reconstructed image at the first resolution, and the reconstructed image is upscaled to a second resolution using the amount of scaling to obtain an upscaled reconstructed image. The second resolution is higher than the first resolution. Loop restoration filtering is applied to the upscaled reconstructed image using loop restoration parameters to obtain a loop restored image at the second resolution.

Claims (57)

1. An apparatus for decoding encoded image data, comprising:

a processor configured to:

decode, from an encoded bitstream, a flag indicating whether an image was encoded using a super-resolution mode, the image encoded at a first resolution;

decode, responsive to the flag indicating that the image was encoded using the super-resolution mode, bits indicating an amount of scaling of the image;

decode, from the encoded bitstream, the image to obtain a reconstructed image at the first resolution;

decode, from the encoded bitstream, data associated with a block of the image that identifies loop restoration parameters;

upscale, using the amount of scaling, the reconstructed image to a second resolution higher than the first resolution to obtain an upscaled reconstructed image; and

apply loop restoration filtering to the upscaled reconstructed image using the loop restoration parameters to obtain a loop restored image at the second resolution.

2. The apparatus of claim 1 , wherein to upscale the reconstructed image comprises to upscale the reconstructed image in only a horizontal dimension.

3. The apparatus of claim 1 , wherein:

to decode the image comprises to decode blocks of the image, the reconstructed image comprising reconstructed blocks, and

a loop restoration unit is associated with more than one of the reconstructed blocks.

4. The apparatus of claim 1 , wherein the data identifies loop restoration parameters for multiple loop restoration units.

5. The apparatus of claim 1 , wherein:

to decode the image comprises to decode blocks of the image, the reconstructed image comprising reconstructed blocks,

the loop restoration parameters are associated with a loop restoration unit of the upscaled reconstructed image, and the processor is configured to:

convert coordinates in the upscaled reconstructed image associated with the loop restoration unit to coordinates in the reconstructed image; and

identify a reconstructed block of the reconstructed image using on the converted coordinates, wherein to decode the loop restoration parameters comprises the loop restoration parameters associated with the loop restoration unit from a portion of the encoded bitstream associated with the identified block.

6. The apparatus of claim 5 , wherein the coordinates in the upscaled reconstructed image correspond to an upper left corner of the loop restoration unit.

7. The apparatus of claim 1 , to upscale the reconstructed image comprises to apply a linear up-scaler to the reconstructed image.

8. The apparatus of claim 7 , wherein to apply the loop restoration filtering comprises to apply Weiner filtering using the loop restoration parameters.

9. A method for decoding encoded image data, comprising:

decoding, from an encoded bitstream, a flag indicating whether an image was encoded using a super-resolution mode, the image encoded at a first resolution;

decoding, responsive to the flag indicating that the image was encoded using the super-resolution mode, bits indicating an amount of scaling of the image;

decoding, from the encoded bitstream, the image to obtain a reconstructed image at the first resolution;

decoding, from the encoded bitstream, loop restoration parameters, wherein decoding the loop restoration parameter comprises decoding, from the encoded bitstream, data associated with one of the blocks of the image identifying the loop restoration parameters;

upscaling, using the amount of scaling, the reconstructed image to a second resolution higher than the first resolution to obtain an upscaled reconstructed image; and

applying loop restoration filtering to the upscaled reconstructed image using the loop restoration parameters to obtain a loop restored image at the second resolution.

10. The method of claim 9 , wherein the image comprises a frame of a video sequence.

11. The method of claim 9 , comprising:

storing, transmitting, or displaying an image based on the loop restored image.

12. The method of claim 9 , wherein the flag is located within a header of the image.

13. The method of claim 9 , comprising:

receiving, from an encoder, the encoded bitstream.

14. An apparatus for encoding image data, comprising:

a processor configured to:

determine loop restoration parameters for an image at a first resolution, wherein the loop restoration parameters are associated with a loop restoration unit of the image;

downscale the image to obtain a downscaled image at a second resolution that is lower that the first resolution;

encode the downscaled image in an encoded bitstream, wherein the downscaled image is partitioned into blocks for encoding;

encode the loop restoration parameters in a portion of the bitstream associated with a block in the encoded bitstream;

encode a flag in a header of the encoded bitstream indicating that the image has been encoded using a super-resolution mode; and

store or transmit the encoded bitstream.

15. The apparatus of claim 14 , wherein the processor is configured to:

decode the downscaled image at the second resolution to obtain a reconstructed image at the second resolution;

upscale the reconstructed image to obtain an upscaled reconstructed image at the first resolution; and

apply loop restoration filtering to the upscaled reconstructed image using the loop restoration parameters.

16. The apparatus of claim 15 , wherein the processor is configured to:

convert coordinates in the image at the first resolution associated with a loop restoration unit to coordinates in the reconstructed image at the second resolution, and

identify a block of the downscaled image based on the converted coordinates, wherein the portion of the bitstream associated with the block corresponds to the identified block of the downscaled image.

17. The apparatus of claim 14 , wherein to downscale the image comprises to downscale the image in only a horizontal dimension.

18. The apparatus of claim 14 , wherein

the loop restoration unit is associated with more than one of the blocks.

19. The apparatus of claim 1 , wherein:

to decode the image comprises to decode blocks of the image, the reconstructed image comprising reconstructed blocks, and

a single reconstructed block is associated with more than one loop restoration unit.

20. The apparatus of claim 14 , wherein:

a single block is associated with more than one loop restoration unit.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 13, 2022
From: JOSHI, URVANG; MUKHERJEE, DEBARGHA; SIMPSON, ANDREW
To: GOOGLE LLC
Reel/Frame 062063/0813 →
Continuity (2)
Continuation 16018105 · Jun 26, 2018
Related Publication 20200228820A1 · Jul 16, 2020
Cited By (1)
US 12,225,192