IP Library Granted Patent US 48,740
Granted Patent E1
US 48,740 · App. 16/460,966 · Granted Sep 14, 2021

Adaptive processing of video streams with reduced color resolution

Inventor: Laurence Alan Thompson (Morgan Hill, CA)
Assignee: LATTICE SEMICONDUCTOR CORPORATION
H04N19/186H04N19/196H04N19/85H04N19/136H04N19/172
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 48,740
App. No.
16/460,966
Granted
Sep 14, 2021
Kind
E1
Abstract

A solution for adaptively processing a digital image with reduced color resolution is described herein. A source device pre-processes a video frame with reduce color resolution by remapping luma components and chroma components of the video frame, and encodes the pre-processed video frame. The source device remaps a half of luma components on a scan line of the video frame onto a data channel of a source line to an encoder and remaps the other half of the luma components on the scan line to another data channel of the source line. The source device remaps the corresponding chroma components onto a third data channel of a source line. By using a data channel conventionally configured to transmit chroma components, the solution enables a video codec to adaptively encode a digital image with reduced color resolution without converting the digital image to full color resolution before the encoding.

Claims (101)

1. A method for encoding a digital image, the method comprising:

receiving the digital image of a plurality of pixels, each pixel having a luma component and two corresponding chroma components;

extracting luma components and chroma components from the plurality of pixels;

reordering the luma components and reordering the chroma components extracted from the plurality of pixels, wherein reordering the luma components and reordering the chroma components comprises:

for a plurality of pixels received on each scan line of the digital image:

providing a first half of the luma components on a first data channel of three data channels, the three data channels configured for transmitting the luma components and chroma components of the plurality of pixels of the scan line;

providing a second half of the luma components on a second data channel of the three data channels; and

providing the two corresponding chroma components on a remaining data channel of the three data channels; and

generating a data structure representing the digital image based on the reordering of the luma components and the reordering of the chroma components.

2. The method of claim 1 , further comprising encoding the digital image according to the data structure.

3. The method of claim 1 , wherein each of the plurality of pixels is in a sampling format, where two chroma components of the pixel are sampled at half the sample rate of luma component of the pixel in horizontal and vertical directions.

4. The method of claim 1 , wherein the plurality of pixels of the digital image are received in an order according to a scan line of the plurality of pixels, each scan lining having a predetermined horizontal resolution.

5. The method of claim 1 , wherein the reordering the luma components comprises:

for the plurality of pixels received on each scan line of the digital image:

separating the luma components of the pixels on the scan line into two parts, wherein each part has a half of the luma components of the plurality of pixels on the scan line.

6. The method of claim 1 , wherein the digital image is partitioned into a plurality of subpictures, each subpicture having a partial horizontal resolution of a corresponding horizontal resolution of the digital image, and each subpicture having a partial vertical resolution of a corresponding vertical resolution of the digital image.

7. The method of claim 1 , further comprising maintaining a spatial relationship of each pixel component within a subpicture after reordering such that the spatial relationship of individual pixel components within the subpicture is the same as the spatial relationship of the corresponding individual components within the subpicture.

8. A method for decoding an encoded digital image, the method comprising:

receiving the encoded digital image of a plurality of pixels, each pixel having a luma component and two corresponding chroma components;

extracting luma components and chroma components from the plurality of pixels;

reordering the luma components and reordering the chroma components extracted from the plurality of pixels according to a data structure describing the reordering, wherein reordering the luma components and reordering the chroma components comprises:

providing the luma components of the plurality of pixels on two data channels of three data channels, the three data channels configured for transmitting the luma components and chroma components of the plurality of pixels of the scan line, wherein each data channel of the two data channels has a half of the luma components of the plurality of pixels on a scan line; and

providing the two corresponding chroma components on a remaining data channel of the three data channels; and

reconstructing a digital image with reduced color resolution based on the reordering of the luma components and the chroma components of the digital image.

9. The method of claim 8 , wherein the reordering the luma components comprises:

reordering the selected luma components on the scan line according to the data structure describing the reordering.

10. A non-transitory computer readable medium storing executable computer program instructions for encoding a digital image, the computer program instructions comprising instructions that when executed cause a computer processor to:

receive the digital image of a plurality of pixels, each pixel having a luma component and two corresponding chroma components;

extract luma components and chroma components from the plurality of pixels;

reorder the luma components and reorder the chroma components extracted from the plurality of pixels, wherein the computer program instructions that when executed cause the computer processor to reorder the luma components and the chroma components further comprise instructions to:

for a plurality of pixels received on each scan line:

provide a first part of the luma components on a first data channel of three data channels, the three data channels configured for transmitting the luma components and chroma components of the plurality of pixels of the scan line;

provide a second part of the luma components on a second data channel of the three data channels; and

provide the two corresponding chroma components on a remaining data channel of the three data channels; and

generate a data structure representing the digital image based on the reordering of the luma components and the reordering of the chroma components.

11. The computer readable medium of claim 10 , further comprising computer program instructions that when executed cause the computer processor to encode the digital image according to the data structure.

12. The computer readable medium of claim 10 , wherein each of the plurality of pixels is in a sampling format, where two chroma components of the pixel are sampled at half the sample rate of luma component of the pixel in horizontal and vertical directions.

13. The computer readable medium of claim 10 , wherein the plurality of pixels of the digital image are received in an order according to a scan line of the plurality of pixels, each scan lining having a predetermined horizontal resolution.

14. The computer readable medium of claim 10 , wherein the computer program instructions for reordering the luma components comprise instructions that when executed cause the computer processor to:

for the plurality of pixels received on each scan line:

separate the luma components of the pixels on the scan line into the two parts, wherein each part has a half of the luma components of the plurality of pixels on the scan line.

15. The computer readable medium of claim of claim 10 , wherein the digital image is partitioned into a plurality of subpictures, each subpicture having a partial horizontal resolution of a corresponding horizontal resolution of the digital image, and each subpicture having a partial vertical resolution of a corresponding vertical resolution of the digital image.

16. The computer readable medium of claim 10 , further comprising computer program instructions to maintain a spatial relationship of each pixel component within a subpicture after reordering such that the spatial relationship of individual pixel components within the subpicture is the same as the spatial relationship of the corresponding individual components within the subpicture.

17. A non-transitory computer readable medium storing executable computer program instructions for decoding an encoded digital image, the computer program instructions comprising instructions that when executed cause a computer processor to:

receive the encoded digital image of a plurality of pixels, each pixel having a luma component and two corresponding chroma components;

extract luma components and chroma components from the plurality of pixels;

reorder the luma components and reorder the chroma components extracted from the plurality of pixels according to a data structure describing the reordering, wherein the computer program instructions for reordering the luma components and reordering the chroma components comprise instructions that when executed cause the computer processor to:

provide the luma components of the pixels on two data channels of three data channels, the three data channels configured for transmitting the luma components and chroma components of the plurality of pixels of the scan line, wherein each data channel of the two data channels has a half of the luma components of the plurality of pixels on a scan line; and

provide the two corresponding chroma components on a remaining data channel of the three data channels; and

reconstruct a digital image with reduced color resolution based on the reordering of the luma components and the chroma components of the digital image.

18. The computer readable medium of claim 17 , wherein the computer program instructions for reordering the luma components comprise instructions that when executed cause the computer processor to:

reorder the selected luma components on the scan line according to the data structure describing the reordering.

19. A source device for encoding a digital image, the device comprising:

a pre-process module configured to:

receive data defining a plurality of pixels of the digital image, wherein each of the plurality of pixels is defined by a luma pixel component and two corresponding chroma pixel components;

separate the luma pixel components and the two corresponding chroma pixel components of the plurality of pixels;

reorder the plurality of pixels from a received order by:

mapping a first portion of the luma pixel components onto a first data channel of three data channels;

mapping a second portion of the luma pixel components onto a second data channel of the three data channels; and

mapping the two corresponding chroma pixel components onto a remaining data channel of the three data channels;

an encoder configured to:

encode the luma pixel components and the two corresponding chroma pixel components on the three data channels to obtain encoded data; and

transmit the encoded data on a transmission channel.

20. The source device of claim 19, wherein the reordering of the plurality of pixels maintains an original relative spatial relationship of the plurality of pixels.

21. The source device of claim 19, wherein the data is associated with a 4:2:2 sampling ratio or a 4:2:0 sampling ratio.

22. The source device of claim 19, wherein the pre-process module is further configured to partition the plurality of pixels into a plurality of subpictures, and wherein each of the plurality of subpictures comprises a subset of the plurality of pixels.

23. The source device of claim 22, wherein each of the plurality of subpictures has a partial horizontal resolution of a corresponding horizontal resolution of the digital image and a partial vertical resolution of a corresponding vertical resolution of the digital image.

24. The source device of claim 22, wherein a first set of the luma pixel components of each of the plurality of subpictures is mapped onto the first data channel, and wherein a second set of the luma pixel components of each of the plurality of subpictures is mapped onto the second data channel.

25. The source device of claim 22, wherein each of the plurality of subpictures contains a set of pixels associated with respective portions of a plurality of scan lines of the digital image.

26. The source device of claim 22, wherein the pre-process module is configured to partition the plurality of pixels by splitting each scan line of the digital image such that the luma pixel components of each scan line is split into a respective first stream and a respective second stream, wherein the luma pixel components of each first stream are mapped onto the first data channel, and wherein the luma pixel components of each second stream are mapped onto the second data channel.

27. The source device of claim 26, wherein, for each scan line, an original relative spatial relationship of the luma pixel components of the first stream is maintained and an original relative spatial relationship of the luma pixel components of the second stream is maintained.

28. The source device of claim 22, wherein each of the plurality of subpictures comprises a portion of the luma pixel components and a portion of only one chroma pixel component of the two corresponding chroma pixel components.

29. The source device of claim 22, wherein the pre-process module is configured to partition the plurality of pixels based on a setting indicative of a number of subpictures and/or a size of each subpicture.

30. The source device of claim 19, wherein the plurality of pixels of the digital image are received in an order according to a scan line of the plurality of pixels, wherein each scan line has a predetermined horizontal resolution.

31. The source device of claim 19, wherein the encoder is configured to transmit information indicative of the reordering to facilitate decoding of the encoded data, wherein the transmission channel is a serial transmission channel.

32. The source device of claim 31, wherein the encoder is configured to encode the luma pixel components and the two corresponding chroma pixel components based on the information, and wherein the serial transmission channel is a high definition multimedia interface (HDMI) channel or a mobile high-definition link (MHL) channel.

33. A sink device for decoding the encoded data from the source device of claim 19, the sink device comprising:

a decoder configured to:

receive the encoded data from the encoder; and

decode the encoded data to obtain decoded data;

a post-process module configured to:

extract the luma pixel components associated with the first and second data channels;

extract the two corresponding chroma pixel components associated with the remaining data channel; and

reconstruct a digital image with reduced color resolution based on the luma pixel components and the two corresponding chroma pixel components.

34. The sink device of claim 33, wherein the post-process module is further configured to:

receive information indicative of the reordering of the plurality of pixels; and

reordering the luma pixel components and the two corresponding chroma pixel components according to the information, wherein the reconstructing of the digital image is based on the reordering of the luma pixel components and the two corresponding chroma pixel components.

35. A sink device for decoding an encoded digital image, the sink device comprising:

a decoder configured to:

receive the encoded digital image comprising a plurality of pixels, wherein each of the plurality of pixels is defined by a luma pixel component and two corresponding chroma pixel components;

decode the encoded digital image to obtain decoded data;

provide the luma pixel components onto two data channels of three data channels;

provide the two corresponding chroma pixel components onto a remaining data channel of the three channels; and

a post-process module configured to:

reorder the luma pixel components and the two corresponding chroma pixel components; and

reconstruct a digital image with reduced color resolution based on the reordering.

36. The sink device of claim 35, wherein the post-process module is configured to reorder by:

reordering the luma pixel components on the two data channels to a luma channel; and

reordering the two corresponding chroma pixel components on the remaining channel to two chroma lines.

37. The sink device of claim 35, wherein each of the two data channels has a respective portion of the luma pixel components, and wherein the reduced color resolution is associated with a 4:2:0 sampling ratio or a 4:2:2 sampling ratio.

38. The sink device of claim 35, wherein the reordering is based on a data structure comprising information indicative of a reordering performed at a source device, and wherein the encoded digital image and the data structure are received via a high definition multimedia interface (HDMI) channel or a mobile high-definition link (MHL) channel.

Assignments (3)
SECURITY INTEREST Recorded Sep 1, 2022
From: LATTICE SEMICONDUCTOR CORPORATION
To: WELLS FARGO BANK, NATIONAL ASSOCIATION, AS ADMINISTRATIVE AGENT
Reel/Frame 061365/0560 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 8, 2019
From: THOMPSON, LAURENCE ALAN
To: SILICON IMAGE, INC.
Reel/Frame 049692/0711 →
MERGER Recorded Jul 8, 2019
From: SILICON IMAGE, INC.
To: LATTICE SEMICONDUCTOR CORPORATION
Reel/Frame 049692/0719 →