IP Library Granted Patent US 12,231,608
Granted Patent B2
US 12,231,608 · App. 17/531,478 · Granted Feb 18, 2025

Multi-view coding with efficient residual handling

Inventors: Heiko Schwarz (Panketal, DE); Thomas Wiegand (Berlin, DE); Martin Winken (Berlin, DE)
Assignee: Dolby Video Compression, LLC
H04N13/128H04N13/161H04N19/463H04N19/597H04N2213/003
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,231,608
App. No.
17/531,478
Granted
Feb 18, 2025
Kind
B2
Abstract

A gain in multi-view coding is achieved as follows: the residual signal involved with coding a dependent view of the multi-view signal is predicted from a reference residual signal of the current picture of the reference view using block-granular disparity-compensated prediction, i.e. using disparity compensated prediction with a disparity defined at, and varying with, block granularity so that each block of the current picture of the dependent view has its own disparity displacement such as its own disparity vector, associated therewith. In other words, a remaining similarity between the residual signal involved with predictively coding the reference view is used in order to predict the residual signal involved with predictively coding the dependent view.

Claims (24)

1. A method for reconstructing a multi-view signal coded in a data stream, comprising:

obtaining a disparity vector representing a view displacement between a current view and a reference view of a multi-view signal;

identifying a reference-view coding block in a picture of the reference view based on the disparity vector;

obtaining reference motion information associated with the reference-view coding block;

determining current motion information associated with a current coding block in a picture of the current view based on the reference motion information; and

reconstructing the current coding block based on the current motion information and a current-view residual signal associated with the current coding block, wherein the current-view residual signal is based on a reference-view residual signal associated with the reference-view coding block.

2. The method of claim 1 , further comprising obtaining the current-view residual signal based on the disparity vector.

3. The method of claim 2 , wherein obtaining the current-view residual signal is based on the disparity vector and the reference-view residual signal associated with the reference-view coding block.

4. The method of claim 3 , wherein the reference-view residual signal and the current-view residual signal are represented in a transform domain.

5. The method of claim 1 , further comprising:

extracting, from the data stream, a disparity vector residual signal;

refining the disparity vector based on the disparity vector residual signal to generate a refined disparity vector; and

identifying the reference-view coding block based on the refined disparity vector.

6. The method of claim 1 , wherein the obtaining of the disparity vector includes estimating the disparity vector based on temporal prediction using information related to a block in a previously reconstructed picture of the current view.

7. A method for encoding a multi-view signal into a data stream, comprising:

obtaining a disparity vector representing a view displacement between a current view and a reference view of a multi-view signal;

identifying a reference-view coding block in a picture of the reference view based on the disparity vector;

determining reference motion information associated with the reference-view coding block based on block-based predictive coding;

determining current motion information associated with a current coding block in a picture of the current view based on the reference motion information; and

inserting into the data stream the disparity vector and the reference motion information, wherein a current-view residual signal associated with the current coding block is based on a reference-view residual signal associated with the reference-view coding block.

8. The method of claim 7 , further comprising determining a reference-view residual signal associated with the reference-view coding block based on the block-based predictive coding; and inserting into the data stream the reference-view residual signal.

9. The method of claim 7 , wherein the current-view residual signal is determined based on the reference-view residual signal and the disparity vector.

10. The method of claim 7 , wherein the reference-view residual signal and the current-view residual signal are represented in a transform domain.

11. The method of claim 7 , wherein the obtaining of the disparity vector includes estimating the disparity vector based on temporal prediction using information related to a block in a previously reconstructed picture of the current view.

Assignments (2)
CHANGE OF NAME Recorded Jan 30, 2026
From: GE VIDEO COMPRESSION, LLC
To: DOLBY VIDEO COMPRESSION, LLC
Reel/Frame 074536/0748 →
CHANGE OF NAME Recorded Nov 26, 2024
From: GE VIDEO COMPRESSION, LLC
To: DOLBY VIDEO COMPRESSION, LLC
Reel/Frame 069450/0615 →
Continuity (5)
Continuation 16848297 · Apr 14, 2020
Continuation 14277850 · May 15, 2014
Continuation PCTEP2012072889 · Nov 16, 2012
Provisional Application 61561452 · Nov 18, 2011
Related Publication 20220159228A1 · May 19, 2022
References Cited (168)
US 7876833B2 · Segall · 2011 [cited by applicant]
US 7885471B2 · Segall · 2011 [cited by applicant]
US 8204129B2 · He · 2012 [cited by applicant]
US 8369406B2 · Kim · 2013 [cited by applicant]
US 8594193B2 · He · 2013 [cited by examiner]
US 10887575B2 · Schwarz et al. · 2021 [cited by applicant]
US 11523098B2 · Schwarz et al. · 2022 [cited by applicant]
US 20030138150A1 · Srinivasan · 2003 [cited by applicant]
US 20050157797A1 · Gaedke · 2005 [cited by applicant]
US 20070019726A1 · Cha · 2007 [cited by applicant]
US 20070147502A1 · Nakamura · 2007 [cited by examiner]
US 20070223582A1 · Borer · 2007 [cited by applicant]
US 20070230567A1 · Wang · 2007 [cited by applicant]
US 20080069247A1 · He · 2008 [cited by applicant]
US 20080089417A1 · Bao · 2008 [cited by applicant]
US 20080165848A1 · Ye · 2008 [cited by applicant]
US 20080170618A1 · Choi · 2008 [cited by applicant]
US 20080198924A1 · Ho · 2008 [cited by examiner]
US 20080205508A1 · Ziauddin · 2008 [cited by applicant]
US 20080211901A1 · Civanlar · 2008 [cited by applicant]
US 20080267291A1 · Vieron · 2008 [cited by applicant]
US 20090010323A1 · Su · 2009 [cited by examiner]
US 20090015662A1 · Kim · 2009 [cited by applicant]
US 20090074061A1 · Yin · 2009 [cited by applicant]
US 20090096643A1 · Chang · 2009 [cited by applicant]
US 20090103616A1 · Ho · 2009 [cited by applicant]
US 20090175349A1 · Ye · 2009 [cited by applicant]
US 20090190662A1 · Park · 2009 [cited by examiner]
US 20090257492A1 · Andersson et al. · 2009 [cited by applicant]
US 20090290643A1 · Yang · 2009 [cited by applicant]
US 20100034260A1 · Shimizu et al. · 2010 [cited by applicant]
US 20100111183A1 · Jeon et al. · 2010 [cited by applicant]
US 20100158129A1 · Lai et al. · 2010 [cited by applicant]
US 20100189177A1 · Shimizu · 2010 [cited by examiner]
US 20100220791A1 · Lin · 2010 [cited by examiner]
US 20100220795A1 · Yin · 2010 [cited by applicant]
US 20100260260A1 · Wiegand · 2010 [cited by applicant]
US 20100260268A1 · Cowan · 2010 [cited by applicant]
US 20100284466A1 · Pandit · 2010 [cited by applicant]
US 20100309292A1 · Ho et al. · 2010 [cited by applicant]
US 20100316139A1 · Le Leannec · 2010 [cited by applicant]
US 20100329342A1 · Joshi · 2010 [cited by applicant]
US 20110038418A1 · Pandit · 2011 [cited by applicant]
US 20110044550A1 · Tian · 2011 [cited by applicant]
US 20110090959A1 · Wiegand · 2011 [cited by applicant]
US 20110142138A1 · Tian · 2011 [cited by applicant]
US 20110222602A1 · Sung et al. · 2011 [cited by applicant]
US 20110286520A1 · Xu · 2011 [cited by applicant]
US 20110317930A1 · Kim · 2011 [cited by applicant]
US 20120023250A1 · Chen · 2012 [cited by applicant]
US 20120075436A1 · Chen · 2012 [cited by applicant]
US 20120082222A1 · Wang · 2012 [cited by applicant]
US 20120114036A1 · Po · 2012 [cited by applicant]
US 20120229602A1 · Chen · 2012 [cited by examiner]
US 20120236115A1 · Zhang · 2012 [cited by applicant]
US 20120269270A1 · Chen · 2012 [cited by applicant]
US 20130229485A1 · Rusanovsky · 2013 [cited by applicant]
US 20130293676A1 · Sugio et al. · 2013 [cited by applicant]
US 20130322536A1 · Yang · 2013 [cited by applicant]
US 20130335522A1 · Zhang · 2013 [cited by examiner]
US 20200036956A1 · Schwarz et al. · 2020 [cited by applicant]
CN 101170702A · 2008 [cited by applicant]
CN 101491096A · 2009 [cited by applicant]
CN 101600108A · 2009 [cited by applicant]
CN 101669367A · 2010 [cited by applicant]
CN 101690220A · 2010 [cited by applicant]
CN 101867813A · 2010 [cited by applicant]
CN 101911700A · 2010 [cited by applicant]
CN 101917619A · 2010 [cited by applicant]
CN 101986716A · 2011 [cited by applicant]
CN 102077246A · 2011 [cited by applicant]
EP 2348733A2 · 2011 [cited by applicant]
IN 101248669A · 2008 [cited by applicant]
JP 200822549A · 2008 [cited by applicant]
JP 2009213161A · 2009 [cited by applicant]
JP 2009543508A · 2009 [cited by applicant]
JP 2010525724A · 2010 [cited by applicant]
JP 201053748A · 2010 [cited by applicant]
JP 2011509631A · 2011 [cited by applicant]
KR 1020090046826A · 2009 [cited by applicant]
KR 102029401B1 · 2019 [cited by applicant]
KR 102090106B1 · 2020 [cited by applicant]
WO 2007037645A1 · 2007 [cited by applicant]
WO 2007081756A2 · 2007 [cited by applicant]
WO 2008007913A1 · 2008 [cited by applicant]
WO 2009005626A2 · 2009 [cited by applicant]
WO 2010043773A1 · 2010 [cited by applicant]
WO 2013072484A1 · 2013 [cited by applicant]
Notice of Allowance issued in corresponding Brazilian Patent Application 11 2014 011425-0 dated Jun. 14, 2022. [cited by applicant]
Notice of Allowance issued in corresponding U.S. Appl. No. 17/129,450 dated Jul. 25, 2022. [cited by applicant]
Office Action issued in corresponding Korean Patent Application No. 10-2021-7034174 dated Feb. 9, 2022, with English translation. [cited by applicant]
Communication pursuant to Article 94(3) EPC issued in corresponding European Patent Application No. 19 195 147.4 dated Apr. 1, 2022. [cited by applicant]
Office Action issued in corresponding Japanese Patent Application 2021-100230 dated Jul. 8, 2022, with English translation. [cited by applicant]
Office Action (Notice of Allowance) issued in corresponding Korean Patent Application No. 10-2021-7034174 dated Oct. 24, 2022, with English translation. [cited by applicant]
Appeal Decision issued in corresponding Japanese Patent Application No. 2019-105062 dated Dec. 13, 2022, with English translation. [cited by applicant]
Notice of Allowance Korean Patent Application No. 10-2018-7024889 dated Jul. 1, 2019. [cited by applicant]
Office Action issued Sep. 17, 2019 in Japanese Patent Application 2017-137033. [cited by applicant]
Office Action issued Dec. 11, 2019 in Korean Application 10-2019-7028653. [cited by applicant]
Notice of Allowance mailed Jan. 14, 2020 in U.S. Appl. No. 14/277,850. [cited by applicant]
Office Action issued Jan. 7, 2020 in Japanese Application 2017-137033. [cited by applicant]
Notice of Allowance mailed Feb. 11, 2020 in U.S. Appl. No. 14/273,730. [cited by applicant]
Notice of Allowance issued in U.S. Appl. No. 16/848,297 dated Jul. 22, 2021. [cited by applicant]
Office Action issued in U.S. Appl. No. 16/848,297 dated Apr. 7, 2021. [cited by applicant]
Final Office Action issued in U.S. Appl. No. 14/277,850 dated Dec. 10, 2018. [cited by applicant]
Final Office Action issued in U.S. Appl. No. 14/277,850 dated Oct. 3, 2017. [cited by applicant]
Office Action issued in U.S. Appl. No. 14/277,850 dated May 30, 2019. [cited by applicant]
Office Action issued in U.S. Appl. No. 14/277,850 dated May 10, 2018. [cited by applicant]
Office Action issued in U.S. Appl. No. 14/277,850 dated Dec. 27, 2016. [cited by applicant]
Notice of Allowance issued in corresponding U.S. Appl. No. 16/871,919 dated Sep. 20, 2021. [cited by applicant]
Office Action issued in corresponding U.S. Appl. No. 16/871,919 dated Jun. 10, 2021. [cited by applicant]
Notice of Allowance issued in corresponding U.S. Appl. No. 16/592,433 dated Sep. 3, 2020. [cited by applicant]
Office Action issued in corresponding European Patent Application No. 19 195 133.4-1207 dated May 2, 2022. [cited by applicant]
Office Action (Notification of Reasons for Refusal) issued in corresponding Japanese Patent Application No. 2019-105062 dated Apr. 11, 2022, with English translation. [cited by applicant]
Office Action issued in corresponding Chinese Patent Application 201910417853.8 dated Sep. 6, 2022. [cited by applicant]
Non-Final Office Action issued in corresponding U.S. Appl. No. 17/129,450 dated Mar. 3, 2022. [cited by applicant]
Office Action issued in corresponding Korean Patent Application No. 10-2020-7007160 dated Jul. 21, 2021. [cited by applicant]
Notice for Decision to Reject issued in corresponding Korean Patent Application No. 10-2020-7007160 dated Apr. 19, 2021. [cited by applicant]
Office Action issued in corresponding Japanese Patent Application No. 2019-105062 dated Feb. 16, 2021 (drafted Feb. 4, 2021), with English translation. [cited by applicant]
Office Action issued in corresponding Indian Patent Application No. 201938021966 dated Feb. 25, 2021. [cited by applicant]
Office Action issued in corresponding U.S. Appl. No. 16/871,866 dated Oct. 9, 2020. [cited by applicant]
Office Action issued Jul. 14, 2020 in Japanese Application 2019-105062. [cited by applicant]
Haitao Yang et al., Regional Disparity Based Motion and Disparity Prediction for MVC, Joint Video Team (JVT) of SO/IEC MPEG & ITU-T VCEG{ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Q.6), 22nd Meeting: Marrakech, Morocco, Jan.… [cited by applicant]
Office Action issued Sep. 30, 2020 in European Application 20168600.3. [cited by applicant]
Notice of Allowance mailed Sep. 3, 2020 in U.S. Appl. No. 16/859,243. [cited by applicant]
Office Action issued Apr. 23, 2020 in European Application 19195147.4. [cited by applicant]
Office Action issued Apr. 23, 2020 in European Application 19195133.4. [cited by applicant]
Office Action issued Jun. 1, 2020 in Korean Application 10-2020-7007160. [cited by applicant]
Bartnik et al., “HEVC Extension for Multiview Video Coding and Multiview Video plus Depth Coding”; ITU-Telecommunications Standardization Sector Study Group 16 Question 6, Document VCEG-AR13, Feb. 3-10, 2012, 42 pages. [cited by applicant]
Ekmekcioglu et al., “Content Adaptive Enhancement of Multi-View Depth Maps for Free Viewpoint Video”, IEEE Journal of Selected Topics in Signal Processing, vol. 5, No. 2, Aprill, 2011, pp. 352-361. [cited by applicant]
ITU-T and ISO/IEC JTC 1, “Advanced Video Coding for Generic Audiovisual Services”, Recommendation ITU-T H.264 and ISO/IEC 14496-10 (MPEG-4 Avc), 2010. [cited by applicant]
Merkle et al., “3D Video Coding: An Overview of Present and Upcoming Standards”, Visual Communications and Image Processing, Jul. 11, 2010, 7 pages, Huang Shan, An Hui, China. [cited by applicant]
Na, S. et al., “Multi-View Depth Video Coding using Depth View Synthesis”, IEEE International Symposium on Circuits and Systems, May 18, 2008, pp. 1400-1403. [cited by applicant]
Office Action dated Feb. 25, 2016 in parallel European Patent Application No. 12784624.4, 4 pages. [cited by applicant]
Office Action mailed Sep. 30, 2016 in U.S. Appl. No. 14/273,730. [cited by applicant]
Official Communication issued in corresponding International Application PCT /EP2012/072889, mailed on Mar. 19, 2013. [cited by applicant]
Official Communication issued in corresponding Korean Patent Application No. 10-2014-7015900 mailed on Jul. 29, 2016. [cited by applicant]
Official Communication issued in International Patent Application No. PCT /EP2012/072299, mailed on May 21, 2013. [cited by applicant]
Schwarz et al., “Description of 30 Video Coding Technology Proposal by Fraunhofer HHI (HEVC compatible, configuration A)”, ISO/IEC JTC1/SC29/WG11, MPEG2011/M22570, Nov. 2011, 46 pages, Geneva, Switzerland. [cited by applicant]
Schwarz, H. et al., “Overview of the Scalable Video Coding Extension of the H.264/AVC Standard”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 17, No. 9, pp. 1103-1120, Sep. 2007. [cited by applicant]
Seo et al., “Motion Information Sharing Mode for Depth Video Coding”, IEEE, 3DTV-Conference: the True Vision-Capture, Transmission and Display of 30 Video, Jun. 7, 2010, pp. 1-4. [cited by applicant]
Shimizu, S. et al., “View Scalable Multiview Video Coding Using 3-D Warping With Depth Map”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 17, No. 11, Nov. 2007, pp. 1485-1495. [cited by applicant]
Stefanoski, N. et al., “Description of 30 Video Coding Technology Proposal by Disney Research Zurich and Fraunhofer HHI”, Motion Picture Expert Group meeting, No. M22668, Nov. 22, 2011, 34 pages. [cited by applicant]
Tanimoto et al., “Multi-view Depth Map of Rena and Akko & Kayo”, ISO/IEC JTC1/SC29/WG11 M14888, Oct. 2007, 5 pages, Shenzhen, China. [cited by applicant]
Tanimoto et al., “View Synthesis Algorithm in View Synthesis Reference Software 2.0 (VSRS2.0)”, ISO/IEC JTCI/SC29/WG11, MPEG 2008/M16090, Feb. 2009,5 pages, Lausanne, Switzerland. [cited by applicant]
Vetro et al., “Overview of the Stereo and Multiview Video Coding Extensions of the H.264/MPEG-4 AVC Standard”, Proceedings of the IEEE, vol. 99, No. 4, Apr. 11, 2011, pp. 626-642. [cited by applicant]
Vetro et al., “Joint Multiview Video Model (JMVM) 6.0”, Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG (ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 0.6, 25th Meeting, Document JVT-Y207, Oct. 2007, 10 pages, Shenzhen, Chi… [cited by applicant]
Zheng et al., “Affine Model for Disparity Estimation in Multiview Sequence Coding”, International Conference on Information Theory and Information Security (ICITIS), Dec. 17, 2010, pp. 736-740, Nanning, China. [cited by applicant]
Office Action mailed Dec. 27, 2016 in U.S. Appl. No. 14/277,850. [cited by applicant]
Office Action issued Feb. 17, 2017 in Chinese Application No. 2012800667391. [cited by applicant]
Office Action issued Mar. 15, 2017 in U.S. Appl. No. 14/272,671. [cited by applicant]
Office Action issued Oct. 20, 2017 in U.S. Appl. No. 14/273,730. [cited by applicant]
Office Action issued Oct. 3, 2017 in U.S. Appl. No. 14/277,850. [cited by applicant]
Office Action issued Nov. 23, 2017 in European Application 12784624.4. [cited by applicant]
Non-final Office Action U.S. Appl. No. 14/277,850 dated May 10, 2018. [cited by applicant]
Konieczny et al., “Depth-Based Inter-view Prediction of Motion Vectors for Improved Multiview Video Coding”, 2010, IEEE. pp. 1-4. [cited by applicant]
Notice of Allowance U.S. Appl. No. 14/272,671 dated Jun. 4, 2018. [cited by applicant]
Non-final Office Action U.S. Appl. No. 14/273,730 dated Jun. 1, 2018. [cited by applicant]
Final Office Action U.S. Appl. No. 14/277,850 dated Dec. 10, 2018. [cited by applicant]
Summons to Attend Oral Hearing EP Application No. 1284624.4 dated Jan. 2, 2019. [cited by applicant]
Final office action U.S. Appl. No. 14/273,730 dated Jan. 28, 2019. [cited by applicant]
Non-final Office Action U.S. Appl. No. 16/120,731 dated Mar. 21, 2019. [cited by applicant]
Non-final Office Action U.S. Appl. No. 14/277,850 dated May 30, 2019. [cited by applicant]
Notice of Allowance U.S. Appl. No. 16/120,731 dated Jul. 3, 2019. [cited by applicant]
Non-final Office Action U.S. Appl. No. 14/273,730 dated Jun. 14, 2019. [cited by applicant]
Decision to Grant issued in corresponding Japanese Patent Application 2021-100230 dated Feb. 14, 2023, with English translation. [cited by applicant]
Communication pursuant to Article 94(3) EPC issued in corresponding European Application No. 20 168 600.3-1208 dated Feb. 13, 2023. [cited by applicant]
Notice of Issuance issued in corresponding Chinese Patent Application No. 201910417853.8 dated Apr. 15, 2023. [cited by applicant]
Office Action issued in corresponding U.S. Appl. No. 17/576,873 dated Mar. 6, 2023. [cited by applicant]