IP Library › Granted Patent US 12,462,815
Granted Patent B2
US 12,462,815 · App. 16/450,682 · Granted Nov 4, 2025

Synchronizing enhanced audio transports with backward compatible audio transports

Inventors: Dipanjan Sen (Dublin, CA); Shankar Thagadur Shivappa (San Diego, CA); Nils Günther Peters (San Diego, CA); Ferdinando Olivieri (San Diego, CA)
Assignee: QUALCOMM Incorporated
G10L19/008G10L21/055H04L65/80
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,462,815
App. No.
16/450,682
Granted
Nov 4, 2025
Kind
B2
Abstract

In general, techniques are described by which to synchronize enhanced audio transports with backward compatible audio transports. A device comprising a memory and one or more processors may be configured to perform the techniques. The memory may store a backward compatible bitstream conforming to a legacy transport format. The processor may obtain, from the backward compatible bitstream, a first audio transport stream, and obtain, from the backward compatible bitstream, a second audio transport stream. The processor(s) may also obtain, from the backward compatible bitstream, indications representative of synchronization information for the first audio transport stream and the second audio transport stream. The processor(s) may synchronize, based on the indications, the first audio transport stream and the second audio transport to obtain synchronized audio data stream. The processor(s) may obtain, based the synchronized audio data, enhanced audio data, and output the enhanced audio data to one or more speakers.

Claims (68)

1 . A method of processing a backward compatible bitstream conforming to a legacy transport format, the method comprising:

obtaining, from the backward compatible bitstream, a first audio transport stream representative of first audio data;

obtaining, from the backward compatible bitstream, a second audio transport stream representative of second audio data;

obtaining, from the backward compatible bitstream, one or more indications identifying synchronization information for one or more of the first audio transport stream and the second audio transport stream, wherein the synchronization information comprises: a first timestamp associated with each of one or more first portions of the first audio transport stream; and a second timestamp associated with each of one or more second portions of the second audio transport stream, and wherein each of the first timestamp and the second timestamp is an eight-bit integer that repeats cyclically;

synchronizing, based on the first timestamps and the second timestamps, each of the one or more first portion of the first audio transport stream and the one or more second portion of the second audio transport stream to obtain a single synchronized audio data stream;

obtaining, based on the single synchronized audio data stream, enhanced audio data; and

outputting the enhanced audio data to one or more speakers.

2 . The method of claim 1 , wherein the first audio data comprises legacy audio data that conforms to the legacy audio format.

3 . The method of claim 2 , wherein legacy audio format comprises one of a monophonic audio format, or a stereo audio format.

4 . The method of claim 2 ,

wherein the second audio data comprises extended audio data that enhances the legacy audio data to obtain the enhanced audio data conforming to an enhanced audio format.

5 . The method of claim 4 , wherein the enhanced audio format comprises one of a 7.1 surround sound format and a 7.1+4H surround sound format.

6 . The method of claim 1 ,

wherein the first portions include first audio frames, and

wherein the second portions include second audio frames.

7 . The method of claim 1 , further comprising receiving, via a transport layer protocol that provides coarse alignment between the first audio transport stream and the second audio transport stream, the backward compatible bitstream.

8 . The method of claim 1 , wherein the legacy transport format comprises a psychoacoustic codec transport format.

9 . The method of claim 8 , wherein the psychoacoustic codec transport format comprises an Advanced Audio Coding (AAC) transport format or an AptX transport format.

10 . The method of claim 1 ,

wherein the legacy audio format comprises an Advanced Audio Coding transport format or an AptX transport format, and

wherein obtaining the enhanced audio data comprises obtaining the enhanced audio data from one or more fill elements specified in accordance with the Advanced Audio Coding transport format or the AptX transport format.

11 . The method of claim 1 , wherein the enhanced audio data represents higher order ambisonic audio data.

12 . The method of claim 1 ,

wherein the second audio data comprises second higher order ambisonic audio data, and

wherein obtaining the enhanced audio data comprises:

de-mixing the first audio data to obtain first higher order ambisonic audio data; and

rendering, based on the first higher order ambisonic audio data and the second higher order ambisonic audio data, the enhanced audio data conforming to an enhanced audio format.

13 . The method of claim 1 , wherein synchronizing each of the one or more first portion of the first audio transport stream and the second portion of the audio transport stream comprises synchronizing, based on the first timestamps and the second timestamps and prior to performing transport format decoding with respect to either of the first audio data and the second audio data, each of the one or more first portion of the first audio transport stream and the one or more second portion of the second audio transport stream to obtain the single synchronized audio data stream.

14 . A method of obtaining a backward compatible bitstream conforming to a legacy transport format, the method comprising:

specifying, in the backward compatible bitstream, a first audio transport stream representative of first audio data;

specifying, in the backward compatible bitstream, a second audio transport stream representative of second audio data;

specifying, in the backward compatible bitstream, one or more indications identifying synchronization information relative to the first audio transport stream and the second audio transport stream, wherein the synchronization information comprises: a first timestamp associated with each of one or more first portions of the first audio transport stream; and a second timestamp associated with each of one or more second portions of the second audio transport stream, and wherein each of the first timestamp and the second timestamp is an eight-bit integer that repeats cyclically; and

outputting the backward compatible bitstream.

15 . A device configured to process a backward compatible bitstream, the device comprising:

one or more memories configured to store at least a portion of the backward compatible bitstream, the backward compatible bitstream conforming to a legacy transport format; and

one or more processors configured to:

obtain, from the backward compatible bitstream, a first audio transport stream representative of first audio data;

obtain, from the backward compatible bitstream, a second audio transport stream representative of second audio data;

obtain, from the backward compatible bitstream, one or more indications representative of synchronization information for one or more of the first audio transport stream and the second audio transport stream, wherein the synchronization information comprises: a first timestamp associated with each of one or more first portions of the first audio transport stream; and a second timestamp associated with each of one or more second portions of the second audio transport stream, and wherein each of the first timestamp and the second timestamp is an eight-bit integer that repeats cyclically;

synchronize, based on the first timestamps and the second timestamps, each of the one or more first portion of the first audio transport stream and the one or more second portion of the second audio transport stream to obtain a single synchronized audio data stream;

obtain, based on the single synchronized audio data stream, enhanced audio data; and

output the enhanced audio data to one or more speakers.

16 . The device of claim 15 , wherein the first audio data comprises legacy audio data that conforms to a legacy audio format.

17 . The device of claim 16 , wherein legacy audio format comprises one of a monophonic audio format, or a stereo audio format.

18 . The device of claim 16 , wherein the second audio data comprises extended audio data that enhances the legacy audio data to obtain enhanced audio data conforming to an enhanced audio format.

19 . The device of claim 18 , wherein the enhanced audio format comprises one of a 7.1 surround sound format and a 7.1+4H surround sound format.

20 . The device of claim 15 ,

wherein the first portions include first audio frames, and

wherein the second portions include second audio frames.

21 . The device of claim 15 , wherein the one or more processors are further configured to receive, via a transport layer protocol that provides coarse alignment between the first audio transport stream and the second audio transport stream, the backward compatible bitstream.

22 . The device of claim 15 , wherein the legacy transport format comprises a psychoacoustic codec transport format.

23 . The device of claim 22 , wherein the psychoacoustic codec transport format comprises an Advanced Audio Coding (AAC) transport format or an AptX transport format.

24 . The device of claim 1 ,

wherein the legacy transport format comprises an Advanced Audio Coding transport format or an AptX transport format, and

wherein the one or more processors are configured to obtain the enhanced audio data from one or more fill elements specified in accordance with the Advanced Audio Coding transport format or the AptX transport format.

25 . The device of claim 15 , wherein the enhanced audio data represents higher order ambisonic audio data.

26 . The device of claim 15 ,

wherein the one or more processors are configured to de-mix the first audio data to obtain first higher order ambisonic audio data,

wherein the second audio data comprises second higher order ambisonic audio data, and

wherein the one or more processors are configured to render, based on the first higher order ambisonic audio data and the second higher order ambisonic audio data, enhanced audio data conforming to an enhanced audio format.

27 . The device of claim 15 , wherein the one or more processors are configured to synchronize, based on the first timestamps and the second timestamps and prior to performing transport format decoding with respect to either of the first audio data and the second audio data, each of the one or more first portion of the first audio transport stream and the one or more second portion of the second audio transport stream to obtain the single synchronized audio data stream.

28 . A device configured to obtain a backward compatible bitstream, the device comprising:

one or more memories configured to store at least a portion of the backward compatible bitstream, the backward compatible bitstream conforming to a legacy transport format; and

one or more processors configured to:

specify, in the backward compatible bitstream, a first audio transport stream representative of first audio data;

specify, in the backward compatible bitstream, a second audio transport stream representative of second audio data;

specify, in the backward compatible bitstream, one or more indications identifying synchronization information relative to the first audio transport stream and the second audio transport stream, wherein the synchronization information comprises: a first timestamp associated with each of one or more first portions of the first audio transport stream; and a second timestamp associated with each of one or more second portions of the second audio transport stream, and wherein each of the first timestamp and the second timestamp is an eight-bit integer that repeats cyclically; and

output the backward compatible bitstream.

Assignments (2)
CORRECTIVE ASSIGNMENT TO CORRECT THE FIRST INVENTORS NAME PREVIOUSLY RECORDED AT REEL: 049907 FRAME: 0006. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Aug 5, 2019
From: SEN, DIPANJAN; THAGADUR SHIVAPPA, SHANKAR; PETERS, NILS GÜNTHER; OLIVIERI, FERDINANDO
To: QUALCOMM INCORPORATED
Reel/Frame 049964/0335 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 30, 2019
From: SENT, DIPANJAN; THAGADUR SHIVAPPA, SHANKAR; PETERS, NILS GÜNTHER; OLIVIERI, FERDINANDO
To: QUALCOMM INCORPORATED
Reel/Frame 049907/0006 →
Continuity (2)
Provisional Application 62693784 · Jul 3, 2018
Related Publication 20200013426A1 · Jan 9, 2020
References Cited (55)
US 6226616B1 · You · 2001 [cited by examiner]
US 9288603B2 · Sen et al. · 2016 [cited by applicant]
US 9609452B2 · Peters et al. · 2017 [cited by applicant]
US 9788133B2 · Sen et al. · 2017 [cited by applicant]
US 9883310B2 · Peters et al. · 2018 [cited by applicant]
US 11212333B1 · Duffy · 2021 [cited by examiner]
US 20060034375A1 · Ward · 2006 [cited by examiner]
US 20070121620A1 · Yang · 2007 [cited by examiner]
US 20070208571A1 · Lemieux · 2007 [cited by examiner]
US 20090046815A1 · Oh et al. · 2009 [cited by applicant]
US 20110110436A1 · Schierl · 2011 [cited by examiner]
US 20110249821A1 · Jaillet et al. · 2011 [cited by applicant]
US 20140016784A1 · Sen · 2014 [cited by examiner]
US 20140016787A1 · Neuendorf et al. · 2014 [cited by applicant]
US 20140126651A1 · Erickson · 2014 [cited by examiner]
US 20150281746A1 · Lam · 2015 [cited by examiner]
US 20150348558A1 · Riedmiller · 2015 [cited by examiner]
US 20180025738A1 · Villemoes et al. · 2018 [cited by applicant]
US 20190007781A1 · Peters et al. · 2019 [cited by applicant]
US 20190349676A1 · Chen · 2019 [cited by examiner]
US 20190392845A1 · Peters et al. · 2019 [cited by applicant]
US 20190392846A1 · Kim et al. · 2019 [cited by applicant]
CN 1357136A · 2002 [cited by applicant]
CN 1976448A · 2007 [cited by applicant]
WO 2018089352A2 · 2018 [cited by applicant]
Dolby Laboratories Inc: “Encoding First-Order Ambisonics with HE-AAC”, 3GPP TSG-SA4 Meeting #95, S4-170937, Belgrade, Serbia, Oct. 9-13, 2017, 7 pages. [cited by applicant]
Peterson J., et al., “Virtual Reality, Augmented Reality, and Mixed Reality Definitions”, EMA, version 1.0, Jul. 7, 2017, 4 Pages. [cited by applicant]
WG11: “Proposed Draft 1.0 of TR: Technical Report on Architectures for Immersive Media”, ISO/IEC JTC1/SC29/WG11/N17685, San Diego, US, Apr. 2018, 14 pages. [cited by applicant]
Audio, “Call for Proposals for 3D Audio,” International Organisation for Standardisation Organisation Internationale de Normalisation ISO/IEC JTC1/SC29/WG11 Coding of Moving Pictures and Audio, ISO/IEC JTC1/SC29/WG11/N1… [cited by applicant]
Audio Subgroup: “White Paper on AAC Transport Formats”, ISO/IEC JTC1/SC29/WG11 N14751, Jul. 2014, 11 pages. [cited by applicant]
Dolby Laboratories Inc: “Draft CR to TR 26.918 on Encoding First-Order Ambisonics with HE-AAC”, 3GPP TSG-SA4 Meeting #96, S4-171234, Albuquerque, NM, USA, Nov. 13-17, 2017, 9 pages. [cited by applicant]
DVB Organization: “TM-AVC1097r9_TS-101-154-v241DTS-UHD.docx ”, DVB, Digital Video Broadcasting, C/O EBU—17A Ancienne Route—CH-1218 Grand Saconnex, Geneva—Switzerland, Jun. 2, 2018 (Jun. 2, 2018), ETSI TS 1 01 154 V2.4.1… [cited by applicant]
ETSI TS 103 589 V1.1.1, “Higher Order Ambisonics (HOA) Transport Format”, Jun. 2018, 33 pages. [cited by applicant]
Fersch C., et al., “Proposed Updates to MPEG-4”, 118. MPEG Meeting; Apr. 3, 2017-Apr. 7, 2017; Hobart; (Motion Picture Expert Group or ISO/IEC JTC1/SC29/WG11), No. m40511, Apr. 5, 2017 (Apr. 5, 2017), pp. 1-20, XP030068… [cited by applicant]
Hellerud E., et al., “Encoding Higher Order Ambisonics with AAC”, Audio Engineering Society—124th Audio Engineering Society Convention 2008, AES, 60 East 42nd Street, Room 2520, New York 10165-2520, USA, May 1, 2008, pp… [cited by applicant]
Herre, et al., “MPEG-H 3D Audio—The New Standard for Coding of Immersive Spatial Audio,” IEEE Journal of Selected Topics in Signal Processing, vol. 9, No. 5, Aug. 2015, pp. 770-779. [cited by applicant]
Hollerweger F., “An Introduction to Higher Order Ambisonic,” Oct. 2008, pp. 13, Accessed online [Jul. 8, 2013] at <URL: flo.mur.at/writings/HOA-intro.pdf>. [cited by applicant]
“Information technology—High Efficiency Coding and Media Delivery in Heterogeneous Environments—Part 3: 3D Audio,” ISO/IEC JTC 1/SC 29/WG11, ISO/IEC 23008-3, 201x(E), Oct. 12, 2016, 797 Pages. [cited by applicant]
“Information technology—High efficiency coding and media delivery in heterogeneous environments—Part 3: Part 3: 3D Audio, Amendment 3: MPEG-H 3D Audio Phase 2,” ISO/IEC JTC 1/SC 29N, ISO/IEC 23008-3:2015/PDAM 3, Jul. 25… [cited by applicant]
“Information technology—High efficiency coding and media delivery in heterogeneous environments—Part 3: 3D Audio,” ISO/IEC JTC 1/SC 29N, Apr. 4, 2014, 337 pp. [cited by applicant]
International Search Report and Written Opinion—PCT/US2019/039029—ISA/EPO—Aug. 20, 2019. [cited by applicant]
Wikipedia: “MP3”, Retrieved from the Internet: https://en.wikipedia.org/wiki/MP3, Nov. 5, 2019, 20 pages. [cited by applicant]
ISO/IEC 14496-3:2009(E), Information technology—Coding of audio-visual objects—Part 3: Audio, Fourth edition Sep. 1, 2009, 1416 pages. [cited by applicant]
ISO/IEC DIS 23008-3 Information Technology—High Efficiency coding and media delivery in heterogeneous environments—Part 3: 3D audio, Jul. 25, 2014 (Jul. 25, 2014), XP055205625, Retrieved from the Internet URL: http://mp… [cited by applicant]
ISO/IEC JTC 1/SC 29 N 14455; ISO/IEC CD 23003-4:2014E; Information technology—MPEG audio technologies—Part 4: Dynamic range control, Apr. 9, 2014, 73 pages. [cited by applicant]
“MPEG-4 Audio Third Edition”, 72. MPEG Meeting; Apr. 18, 2005-Apr. 22, 2005; Susan; (Motion Picture Expert Group or ISO/IEC JTC1/SC29/WG11), No. N7129, Jul. 13, 2005 (Jul. 13, 2005), pp. 1-120, XP030013787, second parag… [cited by applicant]
Poletti M., “Three-Dimensional Surround Sound Systems Based on Spherical Harmonics,” The Journal of the Audio Engineering Society, vol. 53, No. 11, Nov. 2005, pp. 1004-1025. [cited by applicant]
Riedmiller J., et al., “Delivering Scalable Audio Experiences Using AC-4”, IEEE Transactions on Broadcasting, vol. 63, No. 1, Mar. 1, 2017 (Mar. 1, 2017), pp. 179-201, XP055611935, US, ISSN: 0018-9316. DOI: 10.1109/TBC.… [cited by applicant]
Schonefeld V., “Spherical Harmonics,” Jul. 1, 2005, XP002599101, 25 Pages, Accessed online [Jul. 9, 2013] at URL:http://videoarch1.s-inf.de/˜volker/prosem_paper.pdf. [cited by applicant]
Sen D., et al., “RM1-HOA Working Draft Text”, 107. MPEG Meeting; Jan. 13, 2014-Jan. 17, 2014; San Jose; (Motion Picture Expert Group or ISO/IEC JTC1/SC29/WG11), No. m31827, Jan. 11, 2014 (Jan. 11, 2014), 83 Pages, XP030… [cited by applicant]
Sen D., et al., “Technical Description of the Qualcomm's HoA Coding Technology for Phase II”, 109. MPEG Meeting; Jul. 7, 2014-Jul. 11, 2014; Sapporo; (Motion Picture Expert Group or ISO/IEC JTC1/SC29/WG11), No. m34104, … [cited by applicant]
U.S. Appl. No. 16/450,698, filed Jun. 24, 2019, 90 pages. [cited by applicant]
Wikipedia: “Advanced Audio Coding”, Retrieved from the Internet on: https://en.wikipedia.org/wiki/Advanced_Audio_Coding, Nov. 5, 2019, 15 pages. [cited by applicant]
Wikipedia: “Comparison of Audio Coding Formats”, Retrieved from the Internet: https://en.wikipedia.org/wiki/Comparison_of_audio_coding_formats, Nov. 5, 2019, 12 pages. [cited by applicant]
ETSI TS 102 114: “DTS Coherent Acoustics; Core and Extensions with Additional Profiles”, ETSI TS 102 114 V1.6.1 (Aug. 2019), pp. 1-298. [cited by applicant]