IP Library › Granted Patent US 12,382,076
Granted Patent B2
US 12,382,076 · App. 18/636,982 · Granted Aug 5, 2025

Video frame codec architectures

Inventors: Aki Oskari Kuusela (Palo Alto, CA); Ville-Mikko Rautio (Cupertino, CA)
Assignee: Google LLC
H04N19/44H04N19/172H04N19/20H04N19/42H04N19/40
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,382,076
App. No.
18/636,982
Granted
Aug 5, 2025
Kind
B2
Abstract

Techniques and apparatuses are described for video frame codec architectures. A frame decompressor decompresses compressed frames to produce decompressed frames. A frame decompressor controller arbitrates shared access to the frame decompressor. Multiple cores of an SoC request to receive a decompressed frame from the frame decompressor via the frame decompressor controller. The frame decompressor controller can implement a request queue and can order the servicing of requests based on priority of the requests or requesting cores. The frame decompressor controller can also establish a time-sharing protocol for access by the multiple cores. In some implementations, a video decoder is logically integrated with the frame decompressor and stores portions of a decompressed frame in a video buffer, and a display controller retrieves the portions for display using a synchronization mechanism. In analogous manners, a frame compressor controller can arbitrate shared access to a frame compressor for the multiple cores.

Claims (74)

1. An electronic device comprising:

a frame compressor configured to individually compress at a frame-level multiple uncompressed frames to produce multiple compressed frames, the multiple uncompressed frames respectively corresponding to multiple decoded frames;

a frame compressor controller coupled to the frame compressor and configured to arbitrate access to the frame compressor for multiple cores;

a first core of the multiple cores coupled to the frame compressor controller, the first core configured to obtain via the frame compressor controller a compressed frame of the multiple compressed frames produced at the frame-level by the frame compressor; and

a second core of the multiple cores coupled to the frame compressor controller, the second core configured to obtain via the frame compressor controller another compressed frame of the multiple compressed frames produced at the frame-level by the frame compressor.

2. The electronic device of claim 1 , wherein:

the first core is configured to use the compressed frame for a first video-related purpose;

the second core is configured to use the other compressed frame for a second video-related purpose; and

the first video-related purpose and the second video-related purpose comprise different video-related purposes selected from a group of video-related purposes including: video decoding, video encoding, video displaying, video transcoding, or video compensating.

3. The electronic device of claim 1 , wherein:

the first core comprises a video decoder configured to decode a video stream to produce the multiple decoded frames;

the frame compressor is logically integrated with the video decoder; and

the second core comprises a display controller.

4. The electronic device of claim 1 , wherein:

the first core comprises a video decoder configured to decode a video stream to produce the multiple decoded frames; and

the second core comprises a video encoder configured to encode the multiple uncompressed frames into another video stream.

5. The electronic device of claim 1 , wherein the frame compressor controller comprises:

a request queue configured to store multiple requests, each request corresponding to at least one compressed frame; and

a request manager configured to receive the multiple requests from the multiple cores and to insert the multiple requests into the request queue.

6. The electronic device of claim 1 , wherein the frame compressor controller comprises a time-sharing protocol handler configured to establish a time-sharing protocol to share the frame compressor between the multiple cores.

7. The electronic device of claim 6 , wherein the time-sharing protocol handler is configured to at least one of:

establish the time-sharing protocol to include multiple timeslots that are assigned to at least one core of the multiple cores; or

establish the time-sharing protocol to permit reception of an interrupt signal from a core of the multiple cores, the interrupt signal indicative of a request for at least one compressed frame.

8. The electronic device of claim 1 , further comprising:

a frame decompressor configured to individually decompress at the frame-level one or more compressed frames to produce one or more decompressed frames; and

a frame decompressor controller coupled to the frame decompressor and configured to arbitrate access to the frame decompressor for the multiple cores.

9. The electronic device of claim 8 , further comprising:

a video buffer associated with the frame decompressor; and

a synchronization register corresponding to the video buffer, the synchronization register configured to store one or more values indicative of a status of contents of the video buffer, wherein:

the frame decompressor is configured to store in the video buffer at least a portion of at least one decompressed frame of the one or more decompressed frames; and

the frame decompressor controller is configured to write into the synchronization register at least one value indicative of the status of the stored portion.

10. The electronic device of claim 1 , wherein the frame compressor is configured to individually compress at the frame-level the multiple decoded frames to produce at least a portion of the multiple compressed frames.

11. The electronic device of claim 1 , wherein the frame compressor is configured to implement a lossless frame-level compression algorithm to produce the multiple compressed frames respectively from the multiple uncompressed frames.

12. An electronic device comprising:

a frame compressor configured to compress multiple uncompressed frames to produce multiple compressed frames;

a frame compressor controller coupled to the frame compressor and configured to arbitrate access to the frame compressor for multiple cores, the frame compressor controller comprising a time-sharing protocol handler configured to establish a time-sharing protocol to share the frame compressor between the multiple cores, the time-sharing protocol handler configured to at least one of:

establish the time-sharing protocol to include multiple timeslots that are assigned to at least one core of the multiple cores; or

establish the time-sharing protocol to permit reception of an interrupt signal from a core of the multiple cores, the interrupt signal indicative of a request for at least one compressed frame;

a first core of the multiple cores coupled to the frame compressor controller, the first core configured to obtain via the frame compressor controller a compressed frame of the multiple compressed frames produced by the frame compressor; and

a second core of the multiple cores coupled to the frame compressor controller, the second core configured to obtain via the frame compressor controller another compressed frame of the multiple compressed frames produced by the frame compressor.

13. A method for sharing frame compression circuitry between multiple cores, the method comprising:

decoding at least one video stream to produce multiple decoded frames, the multiple decoded frames corresponding to a first uncompressed frame and a second uncompressed frame;

accepting, from a first core, a first request for a first compressed frame;

individually compressing at a frame-level the first uncompressed frame to produce the first compressed frame;

providing, to the first core, the first compressed frame responsive to the first request;

accepting, from a second core, a second request for a second compressed frame;

individually compressing at the frame-level the second uncompressed frame to produce the second compressed frame; and

providing, to the second core, the second compressed frame responsive to the second request.

14. The method of claim 13 , further comprising:

accepting, from a third core, a third request for a third compressed frame;

individually compressing at the frame-level a third uncompressed frame to produce the third compressed frame; and

providing, to the third core, the third compressed frame responsive to the third request, wherein:

the first request is indicative of the first core;

the second request is indicative of the second core; and

the third request is indicative of the third core.

15. The method of claim 13 , further comprising:

establishing a time-sharing protocol for access to the frame compression circuitry by at least the first core and the second core.

16. The method of claim 15 , wherein the establishing comprises:

receiving, during a timeslot that corresponds to the first core, an interrupt from the second core; and

responsive to receipt of the interrupt, switching from performing compression for the first core to performing compression for the second core.

17. The method of claim 13 , further comprising:

performing the providing of the first compressed frame and the providing of the second compressed frame in an order that is at least partially dependent on a relative priority between the first request and the second request.

18. The method of claim 13 , further comprising:

accepting, from at least a portion of the multiple cores, multiple requests for respective ones of multiple decompressed frames;

decompressing multiple compressed frames to respectively produce the multiple decompressed frames; and

providing, to at least the portion of the multiple cores, the multiple decompressed frames responsive to the multiple requests.

19. The method of claim 18 , wherein:

a decompressed frame of the multiple decompressed frames comprises a line and an additional line; and

the method further comprises providing the decompressed frame to a core of the multiple cores by:

polling a synchronization register to obtain at least one value indicative of a read status of the line with respect to a video buffer; and

responsive to the at least one value, storing the additional line into the video buffer.

20. The method of claim 13 , wherein:

the first uncompressed frame comprises or is derived from a first decoded frame of the multiple decoded frames; and

the second uncompressed frame comprises or is derived from a second decoded frame of the multiple decoded frames.

Assignments (2)
CORRECTIVE ASSIGNMENT TO CORRECT THE NATURE OF CONVEYANCE PREVIOUSLY RECORDED AT REEL: 67308 FRAME: 572. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded May 8, 2024
From: KUUSELA, AKI OSKARI; RAUTIO, VILLE-MIKKO
To: GOOGLE LLC
Reel/Frame 067348/0477 →
ENTITY CONVERSION Recorded May 3, 2024
From: KUUSELA, AKI OSKARI; RAUTIO, VILLE-MIKKO
To: GOOGLE LLC
Reel/Frame 067308/0572 →
Continuity (6)
Continuation 17815182 · Jul 26, 2022
Continuation 17651015 · Feb 14, 2022
Division 16870531 · May 8, 2020
Division 15844334 · Dec 15, 2017
Provisional Application 62579820 · Oct 31, 2017
Related Publication 20240267547A1 · Aug 8, 2024
References Cited (75)
US 6058459A · Owen et al. · 2000 [cited by applicant]
US 6157740A · Buerkle et al. · 2000 [cited by applicant]
US 6717989B1 · Simsic et al. · 2004 [cited by applicant]
US 9277218B2 · Guo et al. · 2016 [cited by applicant]
US 9313493B1 · Maaninen · 2016 [cited by applicant]
US 9712830B1 · Horn et al. · 2017 [cited by applicant]
US 10659797B2 · Kuusela et al. · 2020 [cited by applicant]
US 11425404B2 · Kuusela et al. · 2022 [cited by applicant]
US 11962795B2 · Kuusela et al. · 2024 [cited by applicant]
US 11968381B2 · Kuusela et al. · 2024 [cited by applicant]
US 20010007576A1 · Lyu · 2001 [cited by applicant]
US 20050063459A1 · Karaoguz et al. · 2005 [cited by applicant]
US 20060262845A1 · Prakash et al. · 2006 [cited by applicant]
US 20070092004A1 · Ho · 2007 [cited by examiner]
US 20070117015A1 · Yamamoto et al. · 2007 [cited by applicant]
US 20070177015A1 · Arakawa et al. · 2007 [cited by applicant]
US 20080126812A1 · Ahmed et al. · 2008 [cited by applicant]
US 20090257485A1 · Youn · 2009 [cited by applicant]
US 20100034257A1 · Sedeffow et al. · 2010 [cited by applicant]
US 20100166068A1 · Perlman et al. · 2010 [cited by applicant]
US 20100172567A1 · Prokoski · 2010 [cited by applicant]
US 20120093234A1 · Rosenzweig et al. · 2012 [cited by applicant]
US 20150092834A1 · Cote et al. · 2015 [cited by applicant]
US 20180240440A1 · Furihata · 2018 [cited by examiner]
US 20190132604A1 · Kuusela et al. · 2019 [cited by applicant]
US 20200275113A1 · Kuusela et al. · 2020 [cited by applicant]
US 20220166994A1 · Kuusela et al. · 2022 [cited by applicant]
US 20220360803A1 · Kuusela et al. · 2022 [cited by applicant]
CN 102547283 · 2012 [cited by applicant]
CN 105992005 · 2016 [cited by applicant]
EP 1809047A2 · 2007 [cited by examiner]
JP 2007266970 · 2007 [cited by applicant]
JP 2009302704 · 2009 [cited by applicant]
TW 201725907 · 2017 [cited by applicant]
TW 201725908 · 2017 [cited by applicant]
WO 2006121472 · 2006 [cited by applicant]
WO 2019089101 · 2019 [cited by applicant]
“Arm Frame Buffer Compression”, retrieved from https://developer.arm.com/technologies/graphics-technologies/arm-frame-buffer-compression on Sep. 14, 2017, 5 pages. [cited by applicant]
“Final Office Action”, U.S. Appl. No. 17/651,015, Sep. 14, 2023, 10 pages. [cited by applicant]
“First Action Interview Office Action”, U.S. Appl. No. 15/844,334, May 10, 2019, 3 pages. [cited by applicant]
“Foreign Office Action”, KR Application No. 1020197032379, Jan. 21, 2021, 19 pages. [cited by applicant]
“Foreign Office Action”, CN Application No. 201880028788.3, Jan. 30, 2022, 30 pages. [cited by applicant]
“Foreign Office Action”, JP Application No. 2019-559770, Mar. 9, 2021, 13 pages. [cited by applicant]
“Foreign Office Action”, CN Application No. 201880028788.3, Apr. 15, 2021, 31 pages. [cited by applicant]
“Foreign Office Action”, TW Application No. 107119412, Apr. 25, 2019, 9 pages. [cited by applicant]
“Foreign Office Action”, JP Applicaiton No. 2022-063975, May 23, 2023, 11 pages. [cited by applicant]
“Foreign Office Action”, KR Application No. 10-2022-7030753, May 23, 2023, 27 pages. [cited by applicant]
“Foreign Office Action”, EP Application No. 18755951.3, May 27, 2021, 5 pages. [cited by applicant]
“Foreign Office Action”, IN Application No. 201947043036, Jun. 18, 2021, 8 pages. [cited by applicant]
“Foreign Office Action”, KR Application No. 10-2019-7032379, Aug. 12, 2021, 8 pages. [cited by applicant]
“Foreign Office Action”, EP Application No. 18755951.3, Aug. 22, 2022, 7 pages. [cited by applicant]
“Foreign Office Action”, JP Application No. 2019-559770, Oct. 19, 2021, 11 pages. [cited by applicant]
“Foreign Office Action”, KR Application No. 10-2019-7032379, Oct. 27, 2021, 6 pages. [cited by applicant]
“Foreign Office Action”, KR Application No. 10-2021-7032361, Nov. 22, 2021, 23 pages. [cited by applicant]
“Foreign Office Action”, TW Application No. 111105413, Dec. 12, 2023, 7 pages. [cited by applicant]
“International Preliminary Report on Patentability”, Application No. PCT/US2018/042164, May 14, 2020, 15 pages. [cited by applicant]
“International Search Report and Written Opinion”, Application No. PCT/US2018/042164, Oct. 22, 2018, 22 pages. [cited by applicant]
“Non-Final Office Action”, U.S. Appl. No. 17/651,015, Mar. 31, 2023, 8 pages. [cited by applicant]
“Non-Final Office Action”, U.S. Appl. No. 17/815,182, Jul. 3, 2023, 4 pages. [cited by applicant]
“Non-Final Office Action”, U.S. Appl. No. 15/844,334, Aug. 12, 2019, 9 Pages. [cited by applicant]
“Non-Final Office Action”, U.S. Appl. No. 16/870,531, Nov. 23, 2021, 12 pages. [cited by applicant]
“Notice of Allowance”, U.S. Appl. No. 15/844,334, Jan. 14, 2020, 7 Pages. [cited by applicant]
“Notice of Allowance”, U.S. Appl. No. 16/870,531, Apr. 19, 2022, 7 pages. [cited by applicant]
“Notice of Allowance”, U.S. Appl. No. 17/815,182, Dec. 15, 2023, 5 pages. [cited by applicant]
“Notice of Allowance”, U.S. Appl. No. 17/651,015, filed Dec. 18, 2023, 7 pages. [cited by applicant]
“Pre-Interview Communication”, U.S. Appl. No. 15/844,334, May 28, 2019, 4 pages. [cited by applicant]
“Restriction Requirement”, U.S. Appl. No. 16/870,531, Jul. 19, 2021, 6 pages. [cited by applicant]
“Search Report”, TW Application No. 109107219, Oct. 26, 2020, 1 page. [cited by applicant]
“Summons to Attend Oral Proceedings”, EP Application No. 18755951.3, Nov. 22, 2023, 9 pages. [cited by applicant]
Kim, et al., “Multithread Video Coding Processor for the Videophone”, Sep. 1995, 11 pages. [cited by applicant]
Kim, et al., “Single Encoder and Decoder Design for Multi-View Video”, Aug. 2006, 9 pages. [cited by applicant]
Mastronarde, Nicholash. , “A Queuing-Theoretic Approach to Task Scheduling and Processor Selection for Video-Decoding Applications”, Nov. 2007, pp. 1493-1507. [cited by applicant]
Mastronarde, et al., “A Queuing-Theoretic Approach to Task Scheduling and Processor Selection for Video-Decoding Applications”, Nov. 2007, 15 pages. [cited by applicant]
Zhou, et al., “A 530 Mpixels/s 4096×2160@60fps H.264/AVC High Profile Video Decoder Chip”, Apr. 2011, pp. 777-788. [cited by applicant]
“Foreign Office Action”, CN Application No. 202210956690.2, Nov. 25, 2024, 13 pages. [cited by applicant]