IP Library › Granted Patent US 12,231,775
Granted Patent B2
US 12,231,775 · App. 18/734,645 · Granted Feb 18, 2025

Method and apparatus for reconstructing 360-degree image according to projection format

Inventor: Ki Baek Kim (Daejeon, KR)
Assignee: B1 INSTITUTE OF IMAGE TECHNOLOGY, INC.
H04N23/698G06T3/40H04N11/28H04N19/103H04N19/11H04N19/119H04N19/124H04N19/129H04N19/13H04N19/134H04N19/159H04N19/174H04N19/176H04N19/186H04N19/30H04N19/31H04N19/33H04N19/44H04N19/503H04N19/51H04N19/625H04N19/45
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,231,775
App. No.
18/734,645
Granted
Feb 18, 2025
Kind
B2
Abstract

Disclosed are methods and apparatuses for image data encoding/decoding. A method for decoding a 360-degree image includes the steps of: receiving a bitstream obtained by encoding a 360-degree image; generating a prediction image by making reference to syntax information obtained from the received bitstream; adding the generated prediction image to a residual image obtained by dequantizing and inverse-transforming the bitstream, so as to obtain a decoded image; and reconstructing the decoded image into a 360-degree image according to a projection format. Therefore, the performance of image data compression can be improved.

Claims (40)

1. A method for decoding a 360-degree image, the method comprising:

receiving a bitstream in which the 360-degree image is encoded, the bitstream including data of an extended 2-dimensional image, the extended 2-dimensional image including a 2-dimensional image and a predetermined extension region, and the 2-dimensional image being from an image with a 3-dimensional projection structure and including one or more faces;

generating a prediction image by referring to syntax information obtained from the received bitstream;

obtaining a decoded image for the 2-dimensional image by adding the generated prediction image to a residual image, the residual image being obtained by inverse-quantizing and inverse transforming quantized transform coefficients from the bitstream; and

reconstructing the decoded image for the 2-dimensional image into the 360-degree image according to a projection format,

wherein the projection format is selectively determined based on identification information, among a plurality of pre-defined projection formats including an ERP format in which the 360-degree image is projected in a two-dimensional plane or a CMP format in which the 360-degree image is projected in a cube,

wherein a size of the extension region is variably determined based on at least one of first information indicating a width of the extension region or second information indicating a height of the extension region, independently of a size of the 2-dimensional image,

wherein the first information comprises left size information for specifying the width of the extension region on a left side of the face and right size information for specifying the width of the extension region on a right side of the face,

wherein the extension region is not included in the image with the 3-dimensional projection structure,

wherein at least one of the identification information, the first information or the second information is obtained from the bitstream,

wherein the prediction image is generated by selecting one prediction mode among a plurality of prediction modes including intra prediction and inter prediction, and performing prediction based on the selected prediction mode, and

information on the selected prediction mode is obtained from the bitstream.

2. A method for encoding a 360-degree image, the method comprising:

generating a bitstream in which the 360-degree image is encoded, the bitstream including data of an extended 2-dimensional image, the extended 2-dimensional image including a 2-dimensional image and a predetermined extension region, and the 2-dimensional image being from an image with a 3-dimensional projection structure and including one or more faces;

wherein the bitstream comprises syntax information, the syntax information being referred to for generating a prediction image,

the bitstream comprises quantized transform coefficients, the quantized transform coefficients being inverse-quantized and inverse transformed for obtaining a residual image,

the prediction image is added to the residual image for obtaining a decoded image, and

the decoded image is reconstructed into the 360-degree image according to a projection format,

wherein the projection format is selectively determined based on identification information, among a plurality of pre-defined projection formats including an ERP format in which the 360-degree image is projected in a two-dimensional plane or a CMP format in which the 360-degree image is projected in a cube,

wherein a size of the extension region is variably determined based on at least one of first information indicating a width of the extension region or second information indicating a height of the extension region, independently of a size of the 2-dimensional image,

wherein the first information comprises left size information for specifying the width of the extension region on a left side of the face and right size information for specifying the width of the extension region on a right side of the face,

wherein the extension region is not included in the image with the 3-dimensional projection structure,

wherein at least one of the identification information, the first information or the second information is encoded into the bitstream, and

wherein the prediction image is generated by selecting one prediction mode among a plurality of prediction modes including intra prediction and inter prediction, and performing prediction based on the selected prediction mode, and

information on the selected prediction mode is encoded into the bitstream.

3. A method of transmitting a bitstream that is generated by a method for encoding a 360-degree image, comprising:

transmitting the bitstream to an image decoding apparatus;

wherein the method for encoding a 360-degree image comprises:

generating a bitstream in which the 360-degree image is encoded, the bitstream including data of an extended 2-dimensional image, the extended 2-dimensional image including a 2-dimensional image and a predetermined extension region, and the 2-dimensional image being from an image with a 3-dimensional projection structure and including one or more faces;

wherein the bitstream comprises syntax information, the syntax information being referred to for generating a prediction image,

the bitstream comprises quantized transform coefficients, the quantized transform coefficients being inverse-quantized and inverse transformed for obtaining a residual image,

the prediction image is added to the residual image for obtaining a decoded image, and

the decoded image is reconstructed into the 360-degree image according to a projection format,

wherein the projection format is selectively determined based on identification information, among a plurality of pre-defined projection formats including an ERP format in which the 360-degree image is projected in a two-dimensional plane or a CMP format in which the 360-degree image is projected in a cube,

wherein a size of the extension region is variably determined based on at least one of first information indicating a width of the extension region or second information indicating a height of the extension region, independently of a size of the 2-dimensional image,

wherein the first information comprises left size information for specifying the width of the extension region on a left side of the face and right size information for specifying the width of the extension region on a right side of the face,

wherein the extension region is not included in the image with the 3-dimensional projection structure,

wherein at least one of the identification information, the first information or the second information is encoded into the bitstream, and

wherein the prediction image is generated by selecting one prediction mode among a plurality of prediction modes including intra prediction and inter prediction, and performing prediction based on the selected prediction mode, and

information on the selected prediction mode is encoded into the bitstream.

Priority Claims (3)
KR 10-2016-0127878 · Oct 4, 2016 · national
KR 10-2016-0129382 · Oct 6, 2016 · national
KR 10-2017-0090612 · Jul 17, 2017 · national
Continuity (5)
Continuation 18466442 · Sep 13, 2023
Continuation 17487277 · Sep 28, 2021
Continuation 16372237 · Apr 1, 2019
Continuation PCTKR2017011143 · Oct 10, 2017
Related Publication 20240323541A1 · Sep 26, 2024
References Cited (123)
US 6141034A · McCutchen · 2000 [cited by applicant]
US 6157396A · Margulis et al. · 2000 [cited by applicant]
US 7126630B1 · Lee et al. · 2006 [cited by applicant]
US 7292722B2 · Lelescu et al. · 2007 [cited by applicant]
US 7623682B2 · Park et al. · 2009 [cited by applicant]
US 8217988B2 · Park · 2012 [cited by applicant]
US 8264524B1 · Davey · 2012 [cited by applicant]
US 11165958B2 · Kim · 2021 [cited by examiner]
US 20040105597A1 · Lelescu et al. · 2004 [cited by applicant]
US 20060034367A1 · Park · 2006 [cited by applicant]
US 20060034374A1 · Park et al. · 2006 [cited by applicant]
US 20060034529A1 · Park et al. · 2006 [cited by applicant]
US 20060210145A1 · Lee et al. · 2006 [cited by applicant]
US 20060251336A1 · Lelescu et al. · 2006 [cited by applicant]
US 20070133954A1 · Maeda et al. · 2007 [cited by applicant]
US 20110274166A1 · Jeon et al. · 2011 [cited by applicant]
US 20120057788A1 · Fukuhara et al. · 2012 [cited by applicant]
US 20120176367A1 · Genova et al. · 2012 [cited by applicant]
US 20130022113A1 · Chen et al. · 2013 [cited by applicant]
US 20130194255A1 · Lee et al. · 2013 [cited by applicant]
US 20130308709A1 · Norkin et al. · 2013 [cited by applicant]
US 20140146873A1 · Lee et al. · 2014 [cited by applicant]
US 20140184727A1 · Xiao et al. · 2014 [cited by applicant]
US 20140301463A1 · Rusanovskyy et al. · 2014 [cited by applicant]
US 20140307774A1 · Minoo et al. · 2014 [cited by applicant]
US 20140341549A1 · Hattori · 2014 [cited by applicant]
US 20150172544A1 · Deng et al. · 2015 [cited by applicant]
US 20150201204A1 · Chen et al. · 2015 [cited by applicant]
US 20160142697A1 · Budagavi et al. · 2016 [cited by applicant]
US 20160142740A1 · Sharman · 2016 [cited by applicant]
US 20160241837A1 · Cole et al. · 2016 [cited by applicant]
US 20160249056A1 · Tsukuba et al. · 2016 [cited by applicant]
US 20160328824A1 · Kim et al. · 2016 [cited by applicant]
US 20170336705A1 · Zhou et al. · 2017 [cited by applicant]
US 20170339391A1 · Zhou et al. · 2017 [cited by applicant]
US 20170347026A1 · Hannuksela · 2017 [cited by applicant]
US 20170366796A1 · Goldentouch · 2017 [cited by applicant]
US 20180025467A1 · Macmillan et al. · 2018 [cited by applicant]
US 20180027226A1 · Abbas et al. · 2018 [cited by applicant]
US 20180040164A1 · Newman et al. · 2018 [cited by applicant]
US 20180084257A1 · Abbas · 2018 [cited by applicant]
US 20180091735A1 · Wang et al. · 2018 [cited by applicant]
US 20190082161A1 · Cole et al. · 2019 [cited by applicant]
US 20190158849A1 · Yu et al. · 2019 [cited by applicant]
US 20190222862A1 · Shin et al. · 2019 [cited by applicant]
US 20200036997A1 · Li et al. · 2020 [cited by applicant]
US 20200322585A1 · Tsunashima · 2020 [cited by applicant]
US 20220217280A1 · Nishio · 2022 [cited by examiner]
US 20230215018A1 · Lee · 2023 [cited by examiner]
US 20230377163A1 · Metzler · 2023 [cited by examiner]
US 20240048765A1 · Kim · 2024 [cited by applicant]
US 20240334062A1 · Kim · 2024 [cited by examiner]
US 20240340539A1 · Kim · 2024 [cited by examiner]
US 20240406565A1 · Kim · 2024 [cited by examiner]
CN 1174646A · 1998 [cited by applicant]
CN 1209933A · 1999 [cited by applicant]
CN 1759616A · 2006 [cited by applicant]
CN 101010960A · 2007 [cited by applicant]
CN 101479765A · 2009 [cited by applicant]
CN 100544444C · 2009 [cited by applicant]
CN 101641955A · 2010 [cited by applicant]
CN 101888555A · 2010 [cited by applicant]
CN 102611884A · 2012 [cited by applicant]
CN 102918840A · 2013 [cited by applicant]
CN 103250416A · 2013 [cited by applicant]
CN 103621084A · 2014 [cited by applicant]
CN 103907350A · 2014 [cited by applicant]
CN 104396259A · 2015 [cited by applicant]
CN 104584560A · 2015 [cited by applicant]
CN 104602013A · 2015 [cited by applicant]
CN 104662897A · 2015 [cited by applicant]
CN 104811637A · 2015 [cited by applicant]
CN 104833307A · 2015 [cited by applicant]
CN 105247870A · 2016 [cited by applicant]
CN 105554506A · 2016 [cited by applicant]
GB 2548358A · 2017 [cited by applicant]
JP 2015180040A · 2015 [cited by applicant]
KR 1020060015223A · 2006 [cited by applicant]
KR 1020060050350A · 2007 [cited by applicant]
KR 1020070103347A · 2007 [cited by applicant]
KR 1020110118527A · 2011 [cited by applicant]
KR 1020130088085A · 2013 [cited by applicant]
KR 1020150068299A · 2015 [cited by applicant]
KR 1020150113524A · 2015 [cited by applicant]
KR 1020160032909A · 2016 [cited by applicant]
WO WO2013159330A1 · 2013 [cited by applicant]
WO WO2014053518A1 · 2014 [cited by applicant]
WO WO2014084656A1 · 2014 [cited by applicant]
WO WO2015199478A1 · 2015 [cited by applicant]
WO WO2016026457A1 · 2016 [cited by applicant]
ITU-T Recommendation H.365: High efficiency video coding (Apr. 2015) (Year: 2015). [cited by applicant]
Ohm, Jens-Rainer, et al. “Comparison of the coding efficiency of video coding standards—including high efficiency video coding (Hevc).” [cited by applicant]
Sze, Vivienne, et al. “High efficiency video coding (HEVC).” [cited by applicant]
Sjoberg, Rickard, et al. “Overview of HEVC high-level syntax and reference picture management.” [cited by applicant]
Bordes, Philippe, et al. “An overview of the emerging HEVC standard.” [cited by applicant]
Series, H. [cited by applicant]
Auwera, G., M. Coban, and M. Karczewicz. “AHG8: ACP with padding for 360-degree video.” JVET of ITU-T and ISO/IEC. JVET-G0071, Ver 2, Jul. 15, 2017. (pp. 1-11). [cited by applicant]
Bordes, Philippe, et al. “An overview of the emerging HEVC standard.” International Symposium on Signal, Image, Video and Communications, ISIVC, Jul. 2012. (pp. 1-4). [cited by applicant]
Chinese Office Action issued on Jan. 6, 2021, corresponds in Chinese Patent Application No. 201780073622.9 (9 pages in English and 8 pages in Chinese). [cited by applicant]
Choi., Byeongdoo et al., “WD on ISO/IEC 23000-20 Omnidirectional Media Application Format”, [cited by applicant]
Columbia TriStar Home Video, optical disc storing video bitstream of motion picture “Anatomy of a Murder”, 2000. [cited by applicant]
Dong, P.Y., et al. “Re-loss-free intra Coding Using Integer Linear Programming for H. 264/MPEG-4 AVC” Computer Science, vol. 36, 2009. (pp. 1-2). [cited by applicant]
Fuldseth Arild, “Replacing Slices with Titles for High Level Parallelism,” Joint Collaborative Team on Video Coding (JCT-VC-0227) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, Jan. 2011. (pp. 1-5). [cited by applicant]
Fuldseth Arild, “Tiles” Joint Collaborative Team on Video Coding (JCT-VC-F335) of ITU- T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, Jul. 2011. (pp. 1-15). [cited by applicant]
Hanhart, P., et al. “InterDigital's Response to the 360 Video Category in Joint Call for Evidence on Video Compression with Capability beyond HEVC.” JVET of ITU-T and ISO/IEC. JVET-G0024, Ver 2, Jul. 13, 2017. (pp. 1-16… [cited by applicant]
International Search Report issued on Feb. 2, 2018, corresponds in International Patent Application No. PCT/KR2017/011143 (2 pages in English and 2 pages in Korean). [cited by applicant]
ITU-T Recommendation H.265: High efficiency video coding, Apr. 2015. [cited by applicant]
Kim, Jaejoon, et al. “On realization of modified encoding/decoding for high-capacity panoramic video.” 2009 Fourth International Conference on Digital Information Management. IEEE, 2009, (Abstract). [cited by applicant]
Korean Intellectual Property Office issued Written Opinion on Mar. 14, 2022, corresponds in Korean Patent Application No. 10-2021-7034771. [cited by applicant]
Korean Notice of Allowance issued on Nov. 30, 2020, corresponds in Korean Application No. 9-5-2020-083574959 (3 pages in Korean). [cited by applicant]
Korean Notice of Allowance, issued on Nov. 30, 2020, corresponds in Korean Patent Application No. 10-2019-7011755 (3 pages in Korean). [cited by applicant]
Lu, International Organisation for Standardisation Organisation Internationale De Normalisation ISO/IEF JTC1/SC29/WG11 Coding of Moving Pictures and Audio, Tile Segmentation Scheme for OMAF, May-Jun. 2016. (pp. 1-4). [cited by applicant]
Misra, K. et al., “Au Overview of Tiles in HEVC”. 7 IEEE J. of Selected Topics in Signal Processing, Dec. 2013. (pp. 969-977). [cited by applicant]
Ohm, Jens-Rainer, et al. “Comparison of the coding efficiency of video coding standards—including high efficiency video coding (HEVC).” IEEE Transactions on circuits and systems for video technology 22.12, 2012. (pp. 16… [cited by applicant]
“Series H: Audiovisual and Multimedia Systems Infrastructure of Audiovisual Services—Coding of Moving Video: Advanced Video Coding for Generic Audiovisual Services,” ITU-T Telecommunication Standardization Sector of ITU… [cited by applicant]
Series H: Audiovisual and Multimedia Systems Infrastructure of Audiovisual Services—Coding of Moving Video: High Efficiency Video Coding, ITU-T Telecommunication Standardization Sector of ITU, Apr. 2013. (pp. 1-317). [cited by applicant]
Sjoberg, Rickard, et al. “Overview of HEVC high-level syntax and reference picture management.” IEEE transactions on Circuits and Systems for Video Technology 22.12, 2012. (pp. 1858-1870). [cited by applicant]
Sullivan et al. “Overview of the High Efficiency Video Coding (HEVC) Standard”, 22 IEEE Transactions on Circuits & Sys. for Video Tech. Dec. 12, 2012. (pp. 1649-1668). [cited by applicant]
Sullivan, Gary J., et al. “Overview of the high efficiency video coding (HEVC) standard.” IEEE Transactions on circuits and systems for video technology 22.12, 2012. (pp. 1649-1668). [cited by applicant]
Sullivan, Gary, et al. “Meeting report of the 23rd meeting of the Joint Collaborative Team on Video Coding (JCT-VC), San Diego, US, Feb. 19-26, 2016” JCTVC-W_Notes_d7, Joint Collaborative Team on Video Coding (JCT-VC) o… [cited by applicant]
Sze, Vivienne, Madhukar Budagavi, and Gary J. Sullivan. “High efficiency video coding (HEVC).” Integrated circuit and systems, algorithms and architectures. vol. 39. Berlin, Germany: Springer, 2014. (p. 40). [cited by applicant]
Wang, Y.K. “On CMP padding and region-wise packing”, JCT-VC of ITU-T and ISO/IEC. JCTVC-AC0023, Ver.4, Oct. 9, 2017. (pp. 1-15). [cited by applicant]
Zhuo, Li, et al. “Quality Fine Granular Scalability Video Coding Method for Head Shoulder Sequence Images.” Acta Electonica Sinica 32.3, 2004. (pp. 441-445, 5 pages in Japanese, Abstract in English). [cited by applicant]