Frame handling for ML-based upscaling

1. A method for frame handling of an input image in an ML-based upscaling process to produce an output image, the method comprising:

detecting and extracting boxing edge regions from edges of an active picture area of the input image;

extending the extracted boxing edge regions by adding extra pixels around each region of the boxing edge regions, and pixels in each region forming a design,

wherein the added extra pixels form padded edge regions around each region,

wherein the added extra pixels in the active picture area are generated by extending each region into the active picture area, and

wherein the added extra pixels outside the active picture area are generated by transforming the pixels of the design in each region;

upscaling the padded edge regions to produce upscaled padded edge regions;

cropping each of the upscaled padded edge regions to a size corresponding to the size of each of the boxing edge regions prior to the upscaling multiplied by an upscaling factor;

inserting the cropped edge regions into the edges of the output image; and

upscaling a non-edge region of the input image and inserting the upscaled non-edge region into the output image.

2. The method of claim 1 , wherein adding extra pixels around each of the boxing edge regions comprises:

determining how to pad the extra pixels around each of the boxing edge regions; and

adding sub-regions of the extra pixels around each of the boxing edge regions.

3. The method of claim 2 , wherein adding sub-regions comprises determining a type for each of the sub-regions.

4. The method of claim 3 , wherein a first type of the sub-regions comprises

the sub-regions with pixels that are entirely within the active picture area.

5. The method of claim 4 , further comprising

extending each of the boxing edge regions into directions of the sub-regions when the sub-regions are determined to be of the first type.

6. The method of claim 3 , wherein a second type of the sub-regions comprises

the sub-regions with at least one pixel that is not within the active picture area but is on an edge of one of the padded edge regions.

7. The method of claim 6 , further comprising

extending the boxing edge region using a butterfly image over the edges when the sub-regions are determined to be of the second type.

8. The method of claim 3 , wherein a third type of the sub-regions comprises

the sub-regions with at least one pixel that is not within the active picture area but is on a corner of one of the padded edge regions.

9. The method of claim 8 , further comprising

extending each of the boxing edge regions using a butterfly image over the corners when the sub-regions are determined to be of the third type.

10. A system for frame handling of an input image in an ML-based upscaling process to produce an output image, the system comprising:

an edge region extractor configured to detect and extract boxing edge regions from edges of an active picture area of the input image;

an edge region extender configured to extend each region of the boxing edge regions by adding extra pixels around each boxing edge region, and pixels in each region forming a design,

wherein the added extra pixels form padded edge regions around each region,

wherein the added extra pixels in the active picture area are generated by extending each region into the active picture area, and

wherein the added extra pixels outside the active picture area are generated by transforming the pixels of the design in each region,

wherein the edge region extender feeds the padded edge regions to an upscaling processor to upscale and produce upscaled padded edge regions;

an edge region cropper configured to crop each of the upscaled padded edge regions to a size corresponding to the size of each of the boxing edge region prior to the upscaling multiplied by an upscaling factor;

an edge region filler configured to insert the cropped edge regions into the edges of the output image; and

a non-edge region filler configured to upscale a non-edge region of the input image and insert the upscaled non-edge region into the output image.

11. The system of claim 10 , wherein the edge region extender adds the extra pixels around each of the boxing edge regions by adding sub-regions of the extra pixels around each of the boxing edge regions.

12. The system of claim 11 , wherein adding sub-regions comprises determining a type for each of the sub-regions.

13. The system of claim 12 , wherein a first type of the sub-regions comprises

the sub-regions with pixels that are entirely within the active picture area.

14. The system of claim 13 , wherein the edge region extender is further configured to extend each of the boxing edge regions into directions of the sub-regions when the sub-regions are determined to be of the first type.

15. The system of claim 12 , wherein a second type of the sub-regions comprises

the sub-regions with at least one pixel that is not within the active picture area but is on an edge of one of the padded edge regions.

16. The system of claim 15 , wherein the edge region extender is further configured to extend each of the boxing edge regions using a butterfly image over the edges when the sub-regions are determined to be of the second type.

17. The system of claim 12 , wherein a third type of the sub-regions comprises

the sub-regions with at least one pixel that is not within the active picture area but is on a corner of one of the padded edge regions.

18. The system of claim 17 , wherein the edge region extender is configured to extend each of the boxing edge regions using a butterfly image over the corners when the sub-regions are determined to be the third type.

19. A non-transitory computer-readable storage medium storing a computer program to frame handle an input image in an ML-based upscaling process to produce an output image, the computer program comprising executable instructions that cause a computer to:

detect and extract boxing edge regions from edges of an active picture area of the input image;

extend the extracted boxing edge regions by adding extra pixels around each region of the boxing edge regions, and pixels in each region forming a design,

wherein the added extra pixels form padded edge regions around each region,

wherein the added extra pixels in the active picture area are generated by extending each region into the active picture area, and

wherein the added extra pixels outside the active picture area are generated by transforming the pixels of the design in each region;

upscale the padded edge regions to produce upscaled padded edge regions;

crop each of the upscaled padded edge regions to a size corresponding to the size of each of the boxing edge regions prior to the upscaling multiplied by an upscaling factor;

insert the cropped edge regions into the edges of the output image; and

upscale a non-edge region of the input image and inserting the upscaled non-edge region into the output image.

20. The non-transitory computer-readable storage medium of claim 19 , wherein the executable instructions that cause a computer to extend the extracted boxing edge regions comprise executable instructions that cause a computer to:

determine how to pad the extra pixels around each of the boxing edge regions; and

add sub-regions of the extra pixels.