IP Library › Granted Patent US 12,368,850
Granted Patent B2
US 12,368,850 · App. 17/619,946 · Granted Jul 22, 2025

Method of signaling in a video codec

Inventors: Saverio Blasi (London, GB); Andre Seixas Dias (London, GB); Gosala Kulupana (London, GB)
Assignee: British Broadcasting Corporation
H04N19/12H04N19/159H04N19/176H04N19/625
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,368,850
App. No.
17/619,946
Granted
Jul 22, 2025
Kind
B2
Abstract

Encoding of video data in a video codec involves a transform of residuals. This can be composed of a primary transform and a secondary transform. The selection of the secondary transform is effected by considering the characteristics of a block to be encoded. The selection of secondary transform can be signaled to the decoder, or inferred therein.

Claims (27)

1. A decoder for decoding an encoded bitstream representative of a block of a frame of video, the decoder comprising:

an inverse transform module comprising executable instructions that cause one or more processors to:

determine a total number of non-zero coefficients contained in the block to be decoded,

determine a set of candidate inverse secondary transform matrices on the basis of the determined total number of non-zero coefficients contained in the block, wherein the set of candidate inverse secondary transform matrices comprises a plurality of inverse secondary transform matrices, and wherein determining the set of candidate inverse secondary transform matrices comprises determining a candidate number on the basis of the determined total number of non-zero coefficients contained in the block, the candidate number determining how many candidate inverse secondary transform matrices are to be determined in the set of candidate inverse secondary transform matrices,

select an inverse secondary transform matrix from the set of candidate inverse secondary transform matrices based on a signal received on the encoded bitstream, and

apply an inverse matrix transformation to transformed residual information, to extract untransformed residual information, the inverse matrix transformation being governed by the inverse secondary transform matrix and an inverse primary transform matrix; and

an intra-prediction module comprising executable instructions that cause the one or more processors to compute a prediction of the block in accordance with an intra-prediction mode and reconstructing the block by combining the inverse transformed residual data with the prediction.

2. A decoder in accordance with claim 1 , wherein the inverse transform module further causes the one or more processors to determine the set of candidate inverse secondary transform matrices on the basis of whether the block comprises chrominance data or luminance data.

3. A decoder in accordance with claim 1 wherein the inverse transform module further causes the one or more processors to determine the set of candidate inverse secondary transform matrices on the basis of a dimensional characteristic of the block.

4. A decoder in accordance with claim 3 wherein the dimensional characteristic comprises at least one of height or width of the block.

5. A decoder in accordance with claim 1 , wherein the inverse transform module further causes the one or more processors to select the inverse secondary transform matrix on the basis of the selection of a primary transform matrix.

6. A decoder in accordance with claim 1 , wherein the inverse transform module further causes the one or more processors to apply no secondary transform dependent on a primary transform matrix being a predetermined character.

7. A decoder in accordance with claim 6 , wherein the predetermined character of the primary transform matrix comprises that it be derived as an integer approximation of a discrete cosine transform used in the horizontal and vertical directions.

8. A decoder in accordance with claim 7 wherein the discrete cosine transform is DCT2.

9. A method of decoding encoded transformed residual information for a block of a frame of video, the method comprising:

determining a total number of non-zero coefficients contained in the block to be decoded;

determining a set of candidate inverse secondary transform matrices on the basis of the total number of non-zero coefficients contained in the block to be decoded, wherein the set of candidate inverse secondary transform matrices comprises a plurality of inverse secondary transform matrices, and determining the set of candidate inverse secondary transform matrices comprises determining a candidate number on the basis of the determined total number of non-zero coefficients contained in the block, the candidate number determining how many candidate inverse secondary transform matrices are to be determined in the set of candidate inverse secondary transform matrices;

selecting an inverse secondary transform matrix from the set of candidate inverse secondary transform matrices based on a signal received on a bitstream;

inverse transforming the residual information applying an inverse secondary transform and an inverse primary transform, to extract untransformed residual information, wherein the inverse secondary transform is computed on the basis of the inverse secondary transform matrix;

computing a prediction of the current block in accordance with an intra-prediction mode; and

reconstructing the block by combining the inverse transformed residual data with the prediction.

10. A method in accordance with claim 9 , wherein the method further comprises determining the set of candidate inverse secondary transform matrices on the basis of whether the block comprises chrominance data or luminance data.

11. A method in accordance with claim 9 , wherein the method further comprises determining the set of candidate inverse secondary transform matrices on the basis of a dimensional characteristic of the block.

12. A method in accordance with claim 9 , the method further comprising selecting the inverse secondary transform matrix on the basis of the selection of a primary transform matrix.

13. A method in accordance with claim 12 , the method further comprising applying no secondary transform dependent on the primary transform matrix being a predetermined character.

14. A method in accordance with claim 13 wherein the predetermined character of the primary transform matrix comprises that it be derived as an integer approximation of a discrete cosine transform used in the horizontal and vertical directions.

15. A method in accordance with claim 14 wherein the discrete cosine transform is DCT2.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 16, 2021
From: BLASI, SAVERIO; DIAS, ANDRE SEIXAS; KULUPANA, GOSALA
To: BRITISH BROADCASTING CORPORATION
Reel/Frame 058524/0123 →
Priority Claims (1)
GB 1909102 · Jun 25, 2019 · national
Continuity (1)
Related Publication 20220303536A1 · Sep 22, 2022
References Cited (118)
US 4644585A · Crimmins · 1987 [cited by examiner]
US 7515084B1 · Cruz-Albrecht et al. · 2009 [cited by applicant]
US 9609343B1 · Chen et al. · 2017 [cited by applicant]
US 10904522B2 · Lim et al. · 2021 [cited by applicant]
US 11575901B2 · Fan · 2023 [cited by examiner]
US 11689744B2 · Koo · 2023 [cited by examiner]
US 20120008675A1 · Karczewicz et al. · 2012 [cited by applicant]
US 20120314766A1 · Chien et al. · 2012 [cited by applicant]
US 20130051467A1 · Zhou et al. · 2013 [cited by applicant]
US 20130343648A1 · Sato · 2013 [cited by applicant]
US 20170251213A1 · Ye et al. · 2017 [cited by applicant]
US 20170280144A1 · Dvir et al. · 2017 [cited by applicant]
US 20180098081A1 · Zhao et al. · 2018 [cited by applicant]
US 20180249156A1 · Heo et al. · 2018 [cited by applicant]
US 20180302631A1 · Chiang et al. · 2018 [cited by applicant]
US 20190342546A1 · Lin et al. · 2019 [cited by applicant]
US 20190356915A1 · Jang et al. · 2019 [cited by applicant]
US 20210014493A1 · Koo · 2021 [cited by examiner]
US 20210058636A1 · Abe et al. · 2021 [cited by applicant]
US 20210120269A1 · Chen · 2021 [cited by examiner]
US 20220109876A1 · Zhang · 2022 [cited by examiner]
US 20220174299A1 · Zhang · 2022 [cited by examiner]
US 20220312012A1 · Koo · 2022 [cited by examiner]
US 20230037443A1 · Zhang · 2023 [cited by examiner]
CN 108141596A · 2018 [cited by applicant]
CN 110602491 · 2019 [cited by applicant]
DE 102005052702 · 2007 [cited by applicant]
EP 2717578 · 2014 [cited by applicant]
EP 3217663 · 2017 [cited by applicant]
EP 4009632 · 2022 [cited by applicant]
EP 4017000 · 2022 [cited by applicant]
JP 2015213367 · 2015 [cited by applicant]
JP 6998888 · 2022 [cited by applicant]
KR 20180082337 · 2018 [cited by applicant]
RU 2584498 · 2013 [cited by applicant]
RU 2014147451 · 2016 [cited by applicant]
WO 2013039908 · 2013 [cited by applicant]
WO 2016074744 · 2016 [cited by applicant]
WO 2017058615A1 · 2017 [cited by applicant]
WO 2017189048 · 2017 [cited by applicant]
WO WO2018038554A1 · 2018 [cited by applicant]
WO 2018128466 · 2018 [cited by applicant]
WO 2018132380 · 2018 [cited by applicant]
WO 2018127188 · 2018 [cited by applicant]
WO 2018132710 · 2018 [cited by applicant]
WO 2018236031 · 2018 [cited by applicant]
WO WO2019076138A1 · 2019 [cited by applicant]
WO 2019177429 · 2019 [cited by applicant]
WO 2019201232 · 2019 [cited by applicant]
WO 2020009357 · 2020 [cited by applicant]
WO 2020014563 · 2020 [cited by applicant]
WO 2018174402 · 2020 [cited by applicant]
Sharabayko et al., Effectiveness of Internal Block Prediction Methods, Proceedings of Tomsk Polytechnic University, 2013. T. 322. No. 5, with machine translation (14 pages). [cited by applicant]
Bezrukov and Balobanov, “Digital Broadcasting and Applied Television Systems,” Moscow Hot Line—Telecom 2016 (pp. 478, with machine translation (10 pages). [cited by applicant]
Chen et al, Algorithm Description of Joint Exploration Test Model 4, JVET, Document JVET-D1001_v3, 4th Meeting, Chengdu, CN, Oct. 15-21, 2016 (40 pages). [cited by applicant]
Abe et al, CE6: AMT and NSST complexity reduction (CE6-3.3), JVET, Document JVET-K0127-v2, 11th Meeting, Ljubljana, SI, Jul. 10-18, 2018 (6 pages). [cited by applicant]
International Search Report and Written Opinion dated Jul. 9, 2020 in connection with International Application No. PCT/EP2020/061053 (11 pages). [cited by applicant]
Combined Search and Examination Report dated Dec. 4, 2019, for United Kingdom Patent Application No. GB1909102.4 (9 pages). [cited by applicant]
Bross et al, Versatile Video Coding (Draft 5), JVET, Document JVET-N1001-v8, 14th Meeting, Geneva, CH, Mar. 19-27, 2019 (400 pages). [cited by applicant]
Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image Quality Assessment: From Error Visibility to Structural Similarity,” IEEE Transactions on Image Processing, vol. 13, No. 4, Apr. 2004 (14 pages). [cited by applicant]
J. Silva-Martinez: “Wideband Continuous-Time Multi-Bit Delta-Sigma ADCs”, Mar. 31, 2021 (Mar. 31, 2012), XP055624271, Retrieved from the Internet: URL:https://pdfs.semanticscholar.org/eb2c/d8144eca2ac3c8fb24c025ee667ffa… [cited by applicant]
J. Chen, Y. Ye, and S. H. Kim, “Algorithm description for Versatile Video Coding and Test Model 2 (VTM 2),” Tech. Rep., document JVET-K1002, 11th JVET Meeting, Ljubljana, SI, Jul. 2018 (21 pages). [cited by applicant]
G. J. Sullivan, J. Ohm, W. Han, and T. Wiegand, “Overview of the High Efficiency Video Coding (HEVC) Standard,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 22, No. 12, pp. 1649-1668, Dec. 2012 (… [cited by applicant]
T. Laude and J. Ostermann, “Deep Learning-Based Intra Prediction Mode Decision for HEVC,” in 2016 Picture Coding Symposium (PCS), Dec. 2016 (6 pages). [cited by applicant]
T. Wang, M. Chen, and H. Chao, “A Novel Deep Learning-Based Method of Improving Coding Efficiency from the Decoder-end for HEVC,” in 2017 Data Compression Conference (DCC), Apr. 2017, pp. 410-419 (11 pages). [cited by applicant]
J. Pfaff, P. Helle, D. Maniry, S. Kaltenstadler, B. Stallenberger, P. Merkle, M. Siekmann, H. Schwarz, D. Marpe, and T. Wiegand, “Intra Prediction Modes Based on Neural Networks,” Tech. Rep., document JVET-J0037, 10th M… [cited by applicant]
Y. Wang, X. Fan, C. Jia, D. Zhao, and W. Gao, “Neural Network Based Inter Prediction for HEVC,” in 2018 IEEE International Conference on Multimedia and Expo (ICME), Jul. 2018 (6 pages). [cited by applicant]
M. Afonso, F. Zhang, and D. R. Bull, “Video Compression Based on Spatio-Temporal Resolution Adaptation,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 29, No. 1, pp. 275-280, Jan. 2019 (7 pages). [cited by applicant]
T. Li, M. Xu, and X. Deng, “A Deep Convolutional Neural Network Approach for Complexity Reduction on Intra-Mode HEVC,” in 2017 IEEE International Conference on Multimedia and Expo (ICME), Jul. 2017, pp. 1255-1260 (7 pag… [cited by applicant]
N. Westland, A. S. Dias, and M. Mrak, “Decision Trees for Complexity Reduction in Video Compression,” in 2019 IEEE International Conference on Image Processing (ICIP), Sep. 2019 (5 pages). [cited by applicant]
M. Naccari, A. Gabriellini, M. Mrak, S. Blasi, I. Zupancic, and E. Izquierdo, “HEVC Coding Optimisation for Ultra High Definition Television Services,” in 2015 Picture Coding Symposium (PCS), May 2015, pp. 20-24 (5 page… [cited by applicant]
J. Kim, S. Blasi, A. S. Dias, M. Mrak, and E. Izquierdo, “Fast Inter-Prediction Based on Decision Trees for AV1 Encoding,” in 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2… [cited by applicant]
B. Xu, X. Pan, Y. Zhou, Y. Li, D. Yang, and Z. Chen, “CNN-Based Rate-Distortion Modeling for H.265/HEVC,” in 2017 IEEE Visual Communications and Image Processing (VCIP), Dec. 2017 (4 pages). [cited by applicant]
M. Santamaria, E. Izquierdo, S. Blasi, and M. Mrak, “Estimation of Rate Control Parameter s for Video Coding Using CNN,” in 2018 IEEE Visual Communications and Image Processing (VCIP), Dec. 2018 (4 pages). [cited by applicant]
G. Luz, J. Ascenso, C. Brites, and F. Pereira, “Saliency-driven Omnidirectional Imaging Adaptive Coding: Modeling and Assessment,” in 2017 IEEE 19th International Workshop on Multimedia Signal Processing (MMSP), Oct. 20… [cited by applicant]
W. Cui, T. Zhang, S. Zhang, F. Jiang, W. Zuo, Z. Wan, and D. Zhao, “Convolutional Neural Networks Based Intra Prediction for HEVC,” in 2017 Data Compression Conference (DCC), Apr. 2017 (11 pages). [cited by applicant]
R. Yang, M. Xu, T. Liu, Z. Wang, and Z. Guan, “Enhancing Quality for HEVC Compressed Videos,” IEEE Transactions on Circuits and Systems for Video Technology, Sep. 2017 (15 pages). [cited by applicant]
Y. Zhang, T. Shen, X. Ji, Y. Zhang, R. Xiong, and Q. Dai, “Residual Highway Convolutional Neural Networks for In-loop Filtering in HEVC,” IEEE Transactions on Image Processing, vol. 27, No. 8, pp. 3827-3841, Aug. 2018 (… [cited by applicant]
R. Song, D. Liu, H. Li, and F. Wu, “Neural Network-Based Arithmetic Coding of Intra Prediction Modes in HEVC,” in 2017 IEEE Visual Communications and Image Processing (VCIP), Dec. 2017 (4 pages). [cited by applicant]
Y. Li, D. Liu, H. Li, L. Li, F. Wu, H. Zhang, and H. Yang, “Convolutional Neural Network-Based Block Up-Sampling for Intra Frame Coding,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 28, No. 9, F… [cited by applicant]
J. Li, B. Li, J. Xu, and R. Xiong, “Intra Prediction Using Fully Connected Network for Video Coding,” in 2017 IEEE International Conference on Image Processing (ICIP), Sep. 2017 (5 pages). [cited by applicant]
M. Meyer, J. Wiesner, J. Schneider, and C. Rohlfing, “Convolutional Neural Networks for Video Intra Prediction Using Cross-Component Adaptation,” in 2019 IEEE International Conference on Acoustics, Speech and Signal Pro… [cited by applicant]
P. Helle, J. Pfaff, M. Schafer, R. Rischke, H. Schwarz, D. Marpe, and T. Wiegand, “Intra Picture Prediction for Video Coding with Neural Networks,” in 2019 Data Compression Conference (DCC), Mar. 2019, pp. 448-457 (10 p… [cited by applicant]
D.-A. Clevert, T. Unterthiner, and S. Hochreiter, “Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs),” in Proceedings of the International Conference on Learning Representations (ICLR), May 2016… [cited by applicant]
E. Agustsson and R. Timofte, “NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Jul. 2017 (10 pages). [cited by applicant]
J. Boyce, K. Suehring, X. Li, and V. Seregin, “JVET common test conditions and software reference configurations,” Tech. Rep., document JVET-J1010, 10th Meeting, San Diego, US, Apr. 2018 (6 pages). [cited by applicant]
Dhruti Patel et al, “Review on Intra-prediction in High Efficiency Video Coding (HEVC) Standard”, International Journal of Computer Applications, vol. 132, No. 13, pp. 27-30, Dec. 2015 (4 pages). [cited by applicant]
J. Chen et al., “Algorithm Description for Versatile Video Coding and Test Model 5 (VTM 5),” JVET Meeting, Document JVET-N1002, Jun. 2019 (76 pages). [cited by applicant]
J. Pfaff et al., “CE3: Affine Linear Weighted lintra Prediction (CE3-41, CE3-4.2),” JVET Meeting, Document JVET-N0217, Mar. 2019 (18 pages). [cited by applicant]
A. K. Ramasubramonian et al., “Non-CE3: On Signalling of MIP Parameters,” JVET Meeting, Document JVET-00755, Jun. 2019 (6 pages). [cited by applicant]
Y. Li, et al., “A Hybrid Neural Network for Chroma Intra Prediction”, 2018 25th IEEE International Conference on Image Processing (ICIP), IEEE, Oct. 7, 2018, pp. 1797-1801 (6 pages). [cited by applicant]
M. Chiang, C. Hsu, Y. Huang, S. Lei, CE10.1: Combined and Multi-Hypothesis Prediction, Document JVET-K0257-v1, 11th JVET Meeting, Ljubljana, SI, Jul. 2018 (6 pages). [cited by applicant]
W. Xu, H. Yang, Y Zhao, J. Chen, CE10-related: Inter Prediction Sample Filtering, Document JVET-L0375-v1, 12th JVET Meeting, Macao, CN, Oct. 2018 (4 pages). [cited by applicant]
Examination Report issued in connection with Eurasian Patent Appl. No. EA202290141, dated Jul. 18, 2023, and English translation thereof. (17 pages). [cited by applicant]
Bossen F et al, “Non-CE3: A unified luma intra mode list construction process”, No. JVETM0528, Jan. 11, 2019 (Jan. 11, 2019), 13. JVET Meeting; Jan. 9-Jan. 18, 2019; Marrakech; (The Joint Video Exploration Team of ISO/I… [cited by applicant]
Jie Yao et al, “Non-CE3: Intra prediction information coding”, No. JVET-M0210, Jan. 11, 2019 (Jan. 11, 2019), 13. JVET Meeting; Jan. 9-Jan. 18, 2019; Marrakech; (The Joint Video Exploration Team of ISO/IEC JTC1/SC29/WG1… [cited by applicant]
Blasi S et al, “CE3-related: Simplified unified luma intra mode coding”, No. JVET-N0303, Mar. 12, 2019 (Mar. 12, 2019), 14. JVET Meeting; Mar. 19-Mar. 27, 2019; Geneva; (The Joint Video Exploration Team of ISO/IEC JTC1/… [cited by applicant]
International Search Report and Written Opinion dated Mar. 13, 2020, in connection with International Application No. PCT/GB2019/053697 (13 pages). [cited by applicant]
International Preliminary Report on Patentability, issued by the International Bureau on behalf of the International Searching Authority Sep. 23, 2021, in connection with International Patent Application No. PCT/GB2019/… [cited by applicant]
Combined Search and Examination Report dated Sep. 9, 2019, for United Kingdom Patent Application No. GB1903170.7 (3 pages). [cited by applicant]
Combined Search and Examination Report dated Feb. 12, 2020, for United Kingdom Patent Application No. GB1903170.7 (3 pages). [cited by applicant]
International Search Report and Written Opinion dated Jan. 30, 2020, in connection with International Application No. PCT/GB2019/053489 (8 pages). [cited by applicant]
Combined Search and Examination Report dated May 30, 2019, for United Kingdom Patent Application No. GB1820459.4 (3 pages). [cited by applicant]
International Preliminary Report on Patentability, issued by the International Bureau on behalf of the International Searching Authority Jun. 8, 2021, in connection with International Patent Application No. PCT/GB2019/0… [cited by applicant]
Ru-Ling Liao, Chong Soon Lim, CE10.3.1.b: Triangular Prediction Unit Mode, JVET, Document JVET-L0124-v2, 12th Meeting, Macao, CN, Oct. 3-12, 2018 (8 pages). [cited by applicant]
M. Blaser, J. Sauer, and M. R. Wien, Description of SDR and 360° Video Coding Technology Proposal by RWTH Aachen University, JVET, Document JVET-J0023, 10th Meeting, San Diego, US, Apr. 10-20, 2018 (102 pages). [cited by applicant]
Y. Ahn, H. Ryu, D. Sim, Diagonal Motion Partitions on Top of QTBT Block Structure, JVET, Document JVET-H0087, 3th Meeting, Macao, CN, Oct. 18-25, 2017 (6 pages). [cited by applicant]
M. Blaser, J. Sauer, and M. R. Wien, CE10: Results on Geometric Block Partitioning (Test 3.3), JVET, Document JVET-K0146, 11th Meeting, Ljubljana, SI, Jul. 10-18, 2018 (2 pages). [cited by applicant]
J. Chen et al, Algorithm Description for Versatile Video Coding and Test Model 3 (VTM 3), JVET, Document JVET-L1002, 12th Meeting, Macao, CN, Oct. 3-12, 2018 (48 Pages). [cited by applicant]
B. Bross et al, Versatile Video Coding (Draft 3); JVET, Document JVET-L1001-v9, 12th Meeting, Macao, CN, Oct. 3-12, 2018, (238 pages). [cited by applicant]
Combined Search and Examination Report dated Jun. 28, 2019, for United Kingdom Patent Application No. GB1821283.7 (8 pages). [cited by applicant]
International Search Report and Written Opinion dated Jan. 8, 2020, in connection with International Application No. PCT/GB2019/053124 (11 pages). [cited by applicant]
International Preliminary Report on Patentability, issued by the International Bureau on behalf of the International Searching Authority Jun. 16, 2021, in connection with International Patent Application No. PCT/GB2019/… [cited by applicant]
Chinese Office Action issued in connection with CN Patent Application No. 202080046233.9 and machine translation thereof, dated Aug. 1, 2024, 28 pages. [cited by applicant]
Xin Zhao, Description of Core Experiment 6 (CE6): Transforms and transform signalling, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 14th Meeting: Geneva, CH, Mar. 19-27, 2019, 07 pag… [cited by applicant]
Xin Zhao, Description of Core Experiment 6 (CE6): Transforms and transform signalling, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 14th Meeting: Geneva, CH, Mar. 19-27, 2019, 07 pag… [cited by applicant]
Xin Zhao, Description of Core Experiment 6 (CE6): Transforms and transform signalling, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 14th Meeting: Geneva, CH, Mar. 19-27, 2019, 07 pag… [cited by applicant]
Chinese Search Report issued in connection with CN Patent Application No. 202080046233.9 and machine translation thereof, dated Aug. 1, 2024, 06 pages. [cited by applicant]