IP Library › Granted Patent US 12,309,398
Granted Patent B2
US 12,309,398 · App. 17/797,910 · Granted May 20, 2025

Chroma intra prediction in video coding and decoding

Inventors: Marc Gorriz Blanch (London, GB); Marta Mrak (London, GB); Saverio Blasi (London, GB)
Assignee: British Broadcasting Corporation
H04N19/186G06N3/02G06N3/08H04N19/105H04N19/132H04N19/176H04N19/44G06N3/0455G06N3/0464G06N3/0499G06N3/0985
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,309,398
App. No.
17/797,910
Granted
May 20, 2025
Kind
B2
Abstract

An intra-prediction mode is provided for chroma data, in which a neural network implemented attention module directs the encoding of a block of chroma data with respect to luma data for the collocated luma block.

Claims (20)

1. A method of decoding video data, the method comprising:

extracting reference samples from reconstructed luma samples and reconstructed chroma samples; and

constructing at least one block of chroma prediction samples from the reference samples, wherein the constructing of chroma prediction samples depends on the spatial location of the reference samples, and wherein the constructing of chroma prediction samples depends on the usage of an attention module, where the attention module is configured as a deep neural network, wherein the attention module is configured to calculate an attention matrix comprising weights representing a contribution to the prediction of a given output location, each weight corresponding to a reference sample of the extracted reference samples.

2. A method in accordance with claim 1 , wherein the reference samples include reconstructed chroma samples from neighbouring blocks of the block of chroma prediction samples.

3. A method in accordance with claim 1 , wherein the reference samples include reconstructed luma samples collocated with the block of chroma prediction samples.

4. A method in accordance with claim 1 , wherein the reference samples include reconstructed luma samples from the neighbouring blocks of the block of collocated reconstructed luma samples.

5. A method in accordance with claim 1 , wherein the constructing of chroma prediction samples depends on the computation of cross-component information.

6. A method in accordance with claim 1 , wherein the constructing of chroma prediction samples depends on extracting spatial pattern data over a block of luma data using at least one convolutional operation.

7. A method in accordance with claim 1 , comprising controlling, with the attention module, the contribution of each reference neighbouring sample to the computation of the prediction for a sample location.

8. A method in accordance with claim 1 , further implementing one or more other modes of constructing at least one block of chroma data, and wherein the mode of constructing at least one block of chroma data is determined on the basis of a received signal.

9. A decoder for decoding video data, comprising:

a reference sample extractor for extracting reference samples from reconstructed luma samples and reconstructed chroma samples; and

a chroma prediction samples constructor for constructing at least one block of chroma prediction samples from the reference samples, wherein the chroma prediction samples constructor is operable to construct chroma prediction samples depending on the spatial location of the reference samples, wherein the chroma prediction samples constructor is operable to construct chroma prediction samples depending on the usage of an attention module, wherein the attention module is configured as a deep neural network, wherein the attention module is configured to calculate an attention matrix comprising weights representing a contribution to the prediction of a given output location, each weight corresponding to a reference sample of the extracted reference samples.

10. A decoder in accordance with claim 9 , wherein the reference samples include reconstructed chroma samples from neighbouring blocks of the block of chroma prediction samples.

11. A decoder in accordance with claim 9 , wherein the reference samples include reconstructed luma samples collocated with the block of chroma prediction samples.

12. A decoder in accordance with claim 9 , wherein the reference samples include reconstructed luma samples from the neighbouring blocks of the block of collocated reconstructed luma samples.

13. A decoder in accordance with claim 9 , wherein the constructing of chroma prediction samples depends on the computation of cross-component information.

14. A decoder in accordance with claim 9 , wherein the constructing of chroma prediction samples depends on extracting spatial pattern data over a block of luma data using at least one convolutional operation.

15. A decoder in accordance with claim 9 , wherein the attention module is operable to control the contribution of each reference neighbouring sample to the computation of the prediction for a sample location.

16. A decoder in accordance with claim 9 , wherein the chroma prediction samples constructor is operable in one or more other modes of constructing at least one block of chroma data, and wherein the mode of constructing at least one block of chroma data is determined on the basis of a received signal.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 2, 2023
From: BLANCH, MARC GORRIZ; MRAK, MARTA; BLASI, SAVERIO
To: BRITISH BROADCASTING CORPORATION
Reel/Frame 062575/0042 →
Priority Claims (1)
GB 2001722 · Feb 7, 2020 · national
Continuity (1)
Related Publication 20230062509A1 · Mar 2, 2023
References Cited (106)
US 7515084B1 · Cruz-Albrecht et al. · 2009 [cited by applicant]
US 9609343B1 · Chen et al. · 2017 [cited by applicant]
US 10904522B2 · Lim et al. · 2021 [cited by applicant]
US 20120008675A1 · Karczewicz et al. · 2012 [cited by applicant]
US 20120314766A1 · Chien et al. · 2012 [cited by applicant]
US 20130051467A1 · Zhou et al. · 2013 [cited by applicant]
US 20130343648A1 · Sato · 2013 [cited by applicant]
US 20170251213A1 · Ye et al. · 2017 [cited by applicant]
US 20170280144A1 · Dvir · 2017 [cited by applicant]
US 20180249156A1 · Heo et al. · 2018 [cited by applicant]
US 20180302631A1 · Chiang et al. · 2018 [cited by applicant]
US 20190342546A1 · Lin · 2019 [cited by examiner]
US 20190356915A1 · Jang et al. · 2019 [cited by applicant]
US 20190373264A1 · Chong et al. · 2019 [cited by applicant]
US 20200322620A1 · Zhao · 2020 [cited by examiner]
US 20200413049A1 · Biatek et al. · 2020 [cited by applicant]
US 20210058636A1 · Abe · 2021 [cited by applicant]
US 20220070475A1 · Lee et al. · 2022 [cited by applicant]
US 20220109876A1 · Zhang et al. · 2022 [cited by applicant]
US 20220174299A1 · Zhang et al. · 2022 [cited by applicant]
US 20220279162A1 · Ko et al. · 2022 [cited by applicant]
US 20220312012A1 · Koo et al. · 2022 [cited by applicant]
US 20230300347A1 · Kang · 2023 [cited by examiner]
CN 108322723A · 2018 [cited by applicant]
CN 110602491 · 2019 [cited by applicant]
DE 102005052702 · 2007 [cited by applicant]
EP 2717578 · 2014 [cited by applicant]
EP 3217663 · 2017 [cited by applicant]
EP 4009632 · 2022 [cited by applicant]
EP 4017000 · 2022 [cited by applicant]
JP 2015213367 · 2015 [cited by applicant]
JP 6998888 · 2022 [cited by applicant]
KR 20180082337 · 2018 [cited by applicant]
RU 2584498 · 2013 [cited by applicant]
RU 2014147451 · 2016 [cited by applicant]
WO 2013039908 · 2013 [cited by applicant]
WO 2016074744 · 2016 [cited by applicant]
WO 2017058615 · 2017 [cited by applicant]
WO 2017189048 · 2017 [cited by applicant]
WO 2018128466 · 2018 [cited by applicant]
WO 2018132380 · 2018 [cited by applicant]
WO 2018127188 · 2018 [cited by applicant]
WO 2018132710 · 2018 [cited by applicant]
WO 2018236031 · 2018 [cited by applicant]
WO WO2019031410A1 · 2019 [cited by applicant]
WO 2019177429 · 2019 [cited by applicant]
WO 2019201232 · 2019 [cited by applicant]
WO 2020009357 · 2020 [cited by applicant]
WO 2020014563 · 2020 [cited by applicant]
WO 2018174402 · 2020 [cited by applicant]
WO WO2021032045A1 · 2021 [cited by applicant]
Sharabayko et al., Effectiveness of Internal Block Prediction Methods, Proceedings of Tomsk Polytechnic University, 2013. T. 322. No. 5, with machine translation (14 pages). [cited by applicant]
Bezrukov and Balobanov, “Digital Broadcasting and Applied Television Systems,” Moscow Hot Line—Telecom 2016 (p. 478, with machine translation (10 pages). [cited by applicant]
Bossen F et al, “Non-CE3: A unified luma intra mode list construction process”, No. JVETM0528, Jan. 11, 2019 (Jan. 11, 2019), 13. JVET Meeting; Jan. 9, 2019-Jan. 18, 2019; Marrakech; (The Joint Video Exploration Team of… [cited by applicant]
Jie Yao et al, “Non-CE3: Intra prediction information coding”, No. JVET-M0210, Jan. 11, 2019 (Jan. 11, 2019), 13. JVET Meeting; Jan. 9, 2019-Jan. 18, 2019; Marrakech; (The Joint Video Exploration Team of ISO/IEC JTC1/SC… [cited by applicant]
Blasi S et al, “CE3-related: Simplified unified luma intra mode coding”, No. JVET-N0303, Mar. 12, 2019 (Mar. 12, 2019), 14. JVET Meeting; Mar. 19, 2019-Mar. 27, 2019; Geneva; (The Joint Video Exploration Team of ISO/IEC… [cited by applicant]
Ru-Ling Liao, Chong Soon Lim, CE10.3.1.b: Triangular Prediction Unit Mode, JVET, Document JVET-L0124-v2, 12th Meeting, Macao, CN, Oct. 3-12, 2018 (8 pages). [cited by applicant]
M. Blaser, J. Sauer, and M. R. Wien, Description of SDR and 360° Video Coding Technology Proposal by RWTH Aachen University, JVET, Document JVET-J0023, 10th Meeting, San Diego, US, Apr. 10-20, 2018 (102 pages). [cited by applicant]
Y. Ahn, H. Ryu, D. Sim, Diagonal Motion Partitions on Top of QTBT Block Structure, JVET, Document JVET-H0087, 8th Meeting, Macao, CN, Oct. 18-25, 2017 (6 pages). [cited by applicant]
M. Blaser, J. Sauer, and M. R. Wien, CE10: Results on Geometric Block Partitioning (Test 3.3), JVET, Document JVET-K0146, 11th Meeting, Ljubljana, SI, Jul. 10-18, 2018 (2 pages). [cited by applicant]
J. Chen et al, Algorithm Description for Versatile Video Coding and Test Model 3 (VTM 3), JVET, Document JVET-L1002, 12th Meeting, Macao, CN, Oct. 3-12, 2018 (48 Pages). [cited by applicant]
B. Bross et al, Versatile Video Coding (Draft 3); JVET, Document JVET-L1001-v9, 12th Meeting, Macao, CN, Oct. 3-12, 2018, (238 pages). [cited by applicant]
Chen et al, Algorithm Description of Joint Exploration Test Model 4, JVET, Document JVET-D1001_v3, 4th Meeting, Chengdu, CN, Oct. 15-21, 2016 (40 pages). [cited by applicant]
Abe et al, CE6: AMT and NSST complexity reduction (CE6-3.3), JVET, Document JVET-K0127-v2, 11th Meeting, Ljubljana, SI, Jul. 10-18, 2018 (6 pages). [cited by applicant]
Bross et al, Versatile Video Coding (Draft 5), JVET, Document JVET-N1001-v8, 14th Meeting, Geneva, CH, Mar. 19-27, 2019 (400 pages). [cited by applicant]
J. Silva-Martinez: “Wideband Continuous-Time Multi-Bit Delta-Sigma ADCs”, Mar. 31, 2021 (Mar. 31, 2012), XP055624271, Retrieved from the Internet: URL:https://pdfs.semanticscholar.org/eb2c/d8144eca2ac3c8fb24c025ee667ffa… [cited by applicant]
M. Chiang, C. Hsu, Y. Huang, S. Lei, CE10.1: Combined and Multi-Hypothesis Prediction, Document JVET-K0257-v1, 11th JVET Meeting, Ljubljana, SI, Jul. 2018 (6 pages). [cited by applicant]
W. Xu, H. Yang, Y Zhao, J. Chen, CE10-related: Inter Prediction Sample Filtering, Document JVET-L0375-v1, 12th JVET Meeting, Macao, CN, Oct. 2018 (4 pages). [cited by applicant]
J. Chen, Y. Ye, and S. H. Kim, “Algorithm description for Versatile Video Coding and Test Model 2 (VTM 2),” Tech. Rep., document JVET-K1002, 11th JVET Meeting, Ljubljana, SI, Jul. 2018 (21 pages). [cited by applicant]
G. J. Sullivan, J. Ohm, W. Han, and T. Wiegand, “Overview of the High Efficiency Video Coding (HEVC) Standard,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 22, No. 12, pp. 1649-1668, Dec. 2012 (… [cited by applicant]
T. Laude and J. Ostermann, “Deep Learning-Based Intra Prediction Mode Decision for HEVC,” in 2016 Picture Coding Symposium (PCS), Dec. 2016 (6 pages). [cited by applicant]
T. Wang, M. Chen, and H. Chao, “A Novel Deep Learning-Based Method of Improving Coding Efficiency from the Decoder-end for HEVC,” in 2017 Data Compression Conference (DCC), Apr. 2017, pp. 410-419 (11 pages). [cited by applicant]
J. Pfaff, P. Helle, D. Maniry, S. Kaltenstadler, B. Stallenberger, P. Merkle, M. Siekmann, H. Schwarz, D. Marpe, and T. Wiegand, “Intra Prediction Modes Based on Neural Networks,” Tech. Rep., document JVET-J0037, 10th M… [cited by applicant]
Y. Wang, X. Fan, C. Jia, D. Zhao, and W. Gao, “Neural Network Based Inter Prediction for HEVC,” in 2018 IEEE International Conference on Multimedia and Expo (ICME), Jul. 2018 (6 pages). [cited by applicant]
M. Afonso, F. Zhang, and D. R. Bull, “Video Compression Based on Spatio-Temporal Resolution Adaptation,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 29, No. 1, pp. 275-280, Jan. 2019 (7 pages). [cited by applicant]
T. Li, M. Xu, and X. Deng, “A Deep Convolutional Neural Network Approach for Complexity Reduction on Intra-Mode HEVC,” in 2017 IEEE International Conference on Multimedia and Expo (ICME), Jul. 2017, pp. 1255-1260 (7 pag… [cited by applicant]
N. Westland, A. S. Dias, and M. Mrak, “Decision Trees for Complexity Reduction in Video Compression,” in 2019 IEEE International Conference on Image Processing (ICIP), Sep. 2019 (5 pages). [cited by applicant]
M. Naccari, A. Gabriellini, M. Mrak, S. Blasi, I. Zupancic, and E. Izquierdo, “HEVC Coding Optimisation for Ultra High Definition Television Services,” in 2015 Picture Coding Symposium (PCS), May 2015, pp. 20-24 (5 page… [cited by applicant]
J. Kim, S. Blasi, A. S. Dias, M. Mrak, and E. Izquierdo, “Fast Inter-Prediction Based on Decision Trees for AV1 Encoding,” in 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2… [cited by applicant]
B. Xu, X. Pan, Y. Zhou, Y. Li, D. Yang, and Z. Chen, “CNN-Based Rate-Distortion Modeling for H.265/HEVC,” in 2017 IEEE Visual Communications and Image Processing (VCIP), Dec. 2017 (4 pages). [cited by applicant]
M. Santamaria, E. Izquierdo, S. Blasi, and M. Mrak, “Estimation of Rate Control Parameter s for Video Coding Using CNN,” in 2018 IEEE Visual Communications and Image Processing (VCIP), Dec. 2018 (4 pages). [cited by applicant]
G. Luz, J. Ascenso, C. Brites, and F. Pereira, “Saliency-driven Omnidirectional Imaging Adaptive Coding: Modeling and Assessment,” in 2017 IEEE 19th International Workshop on Multimedia Signal Processing (MMSP), Oct. 20… [cited by applicant]
W. Cui, T. Zhang, S. Zhang, F. Jiang, W. Zuo, Z. Wan, and D. Zhao, “Convolutional Neural Networks Based Intra Prediction for HEVC,” in 2017 Data Compression Conference (DCC), Apr. 2017 (11 pages). [cited by applicant]
R. Yang, M. Xu, T. Liu, Z. Wang, and Z. Guan, “Enhancing Quality for HEVC Compressed Videos,” IEEE Transactions on Circuits and Systems for Video Technology, Sep. 2017 (15 pages). [cited by applicant]
Y. Zhang, T. Shen, X. Ji, Y. Zhang, R. Xiong, and Q. Dai, “Residual Highway Convolutional Neural Networks for In-loop Filtering in HEVC,” IEEE Transactions on Image Processing, vol. 27, No. 8, pp. 3827-3841, Aug. 2018 (… [cited by applicant]
R. Song, D. Liu, H. Li, and F. Wu, “Neural Network-Based Arithmetic Coding of Intra Prediction Modes in HEVC,” in 2017 IEEE Visual Communications and Image Processing (VCIP), Dec. 2017 (4 pages). [cited by applicant]
Y. Li, D. Liu, H. Li, L. Li, F. Wu, H. Zhang, and H. Yang, “Convolutional Neural Network-Based Block Up-Sampling for Intra Frame Coding,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 28, No. 9, F… [cited by applicant]
J. Li, B. Li, J. Xu, and R. Xiong, “Intra Prediction Using Fully Connected Network for Video Coding,” in 2017 IEEE International Conference on Image Processing (ICIP), Sep. 2017 (5 pages). [cited by applicant]
M. Meyer, J. Wiesner, J. Schneider, and C. Rohlfing, “Convolutional Neural Networks for Video Intra Prediction Using Cross-Component Adaptation,” in 2019 IEEE International Conference on Acoustics, Speech and Signal Pro… [cited by applicant]
P. Helle, J. Pfaff, M. Schafer, R. Rischke, H. Schwarz, D. Marpe, and T. Wiegand, “Intra Picture Prediction for Video Coding with Neural Networks,” in 2019 Data Compression Conference (DCC), Mar. 2019, pp. 448-457 (10 p… [cited by applicant]
D.-A. Clevert, T. Unterthiner, and S. Hochreiter, “Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs),” in Proceedings of the International Conference on Learning Representations (ICLR), May 2016… [cited by applicant]
E. Agustsson and R. Timofte, “NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Jul. 2017 (10 pages). [cited by applicant]
J. Boyce, K. Suehring, X. Li, and V. Seregin, “JVET common test conditions and software reference configurations,” Tech. Rep., document JVET-J1010, 10th Meeting, San Diego, US, Apr. 2018 (6 pages). [cited by applicant]
Dhruti Patel et al, “Review on Intra-prediction in High Efficiency Video Coding (HEVC) Standard”, International Journal of Computer Applications, vol. 132, No. 13, pp. 27-30, Dec. 2015 (4 pages). [cited by applicant]
J. Chen et al., “Algorithm Description for Versatile Video Coding and Test Model 5 (VTM 5),” JVET Meeting, Document JVET-N1002, Jun. 2019 (76 pages). [cited by applicant]
A. K. Ramasubramonian et al., “Non-CE3: On Signalling of MIP Parameters,” JVET Meeting, Document JVET-O0755, Jun. 2019 (6 pages). [cited by applicant]
Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image Quality Assessment: From Error Visibility to Structural Similarity,” IEEE Transactions on Image Processing, vol. 13, No. 4, Apr. 2004 (14 pages). [cited by applicant]
Examination Report issued in connection with Eurasian Patent Appl. No. EA202292258, dated Mar. 24, 2023, and English translation thereof (20 pages). [cited by applicant]
Examnation Report issued in connection with Patent Appl. No. GB2001722.4, dated Feb. 13, 2023 (4 pages). [cited by applicant]
International Search Report of the International Searching Authority issued in connection with International Application No. PCT/GB2020/052963, Mar. 26, 2021 (4 pages). [cited by applicant]
Written Opinion of the International Searching Authority issued in connection with International Application No. PCT/GB2020/052963, Mar. 26, 2021 (5 pages). [cited by applicant]
Y. Li, et al., “A Hybrid Neural Network for Chroma Intra Prediction”, 2018 25th IEEE International Conference on Image Processing (ICIP), IEEE, Oct. 7, 2018, pp. 1797-1801 (6 pages). [cited by applicant]
International Preliminary Report on Patentability of the International Bureau issued in connection with International Patent Application No. PCT/GB2020/052963, Aug. 18, 2022 (7 pages). [cited by applicant]
Combined Search and Examination Report issued in connection with United Kingdom Patent Application No. GB2001722.4, Aug. 6, 2020 (7 pages). [cited by applicant]
J. Pfaff et al., “CE3: Affine Linear Weighted Intra Prediction (CE3-41, CE3-4.2),” JVET Meeting, Document JVET-N0217, Mar. 2019 (18 pages). [cited by applicant]
Chinese Office Action issued in connection with CN Patent Application No. 202080099291.8 and machine translation thereof, dated Dec. 11, 2024, 19 pages. [cited by applicant]
Cited By (1)
US 12,452,463