IP Library › Granted Patent US 12,250,260
Granted Patent B2
US 12,250,260 · App. 18/480,988 · Granted Mar 11, 2025

Supporting quality of service for media communications

Inventors: Imed Bouazizi (Frisco, TX); Thomas Stockhammer (Bergen, DE); Nikolai Konrad Leung (San Francisco, CA)
Assignee: QUALCOMM INCORPORATED
H04L65/80H04L65/1069H04L65/1108
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,250,260
App. No.
18/480,988
Granted
Mar 11, 2025
Kind
B2
Abstract

A client device (e.g., user equipment or “UE”) may be configured to engage in a media communication session, such as a WebRTC session, with another client device. The client devices may separate a quality of service (QoS) specification from a QoS flow definition, to allow for separate interactive connectivity establishment (ICE) negotiation. The QoS specification may cover all segments of a connection for the media communication session. For example, QoS may be requested for a case where a server (e.g., a Traversal Using Relay Network Address Translation (TURN) server) is hosted by a mobile network operator (MNO). The QoS specification and the QoS flow description may be linked.

Claims (60)

1. A method of applying quality of service to a media communication session, the method comprising:

determining, by a first client device communicatively coupled to a first mobile network operator (MNO), a list of interactive connectivity establishment (ICE) candidates for a second client device;

determining, by the first client device, valid ICE candidates in the list of ICE candidates for the second client device;

for one or more of the valid ICE candidates, sending, by the first client device, data representing quality of service (QOS) flows associated with the one or more of the valid ICE candidates to a server device executing an application function (AF), the AF providing media control for the media communication session;

determining, by the first client device, one of the valid ICE candidates and one of the QoS flows associated with the one of the valid ICE candidates;

establishing, by the first client device, the media communication session with the second client device using the determined one of the valid ICE candidates;

determining, by the first client device, an association between a QoS specification and the one of the QoS flows for the media communication session; and

invoking QoS provided by the first MNO using a service based architecture (SBA) procedure offered by the first MNO.

2. The method of claim 1 , wherein the second client device is communicatively coupled to a second MNO, and determining the list of ICE candidates comprises receiving data representing one or more of the ICE candidates from the first MNO.

3. The method of claim 1 , wherein the second client device is communicatively coupled to the first MNO.

4. The method of claim 1 , further comprising receiving, from the server device executing the AF, data representative of at least one of a Traversal Using Relay Network Address Translation (TURN) server or a Session Traversal of UDP (STUN) server.

5. The method of claim 1 , wherein the data representing the QOS flows associated with the one or more of the valid ICE candidates includes data representing associations between QoS flow descriptions and QoS specifications.

6. The method of claim 1 , wherein the data representing the QoS flows associated with the one or more of the valid ICE candidates includes data describing a service data flow for which QoS is provided.

7. The method of claim 1 , wherein the data representing the QoS flows associated with the one or more of the valid ICE candidates includes a reference to the QoS specification.

8. The method of claim 1 , further comprising sending an update message to the server device executing the AF including data associating one of the valid ICE candidates with one of the QoS flows.

9. The method of claim 1 , wherein the media communication session comprises a Web Real-Time Communication (WebRTC) communication session.

10. A first client device for applying quality of service to a media communication session, the first client device being communicatively coupled to a first mobile network operator (MNO), the first client device comprising:

a memory configured to store media data; and

a processing system implemented in circuitry and configured to:

determine a list of interactive connectivity establishment (ICE) candidates for a second client device;

determine valid ICE candidates in the list of ICE candidates;

for one or more of the valid ICE candidates, send data representing quality of service (QOS) flows associated with the one or more of the valid ICE candidates to a server device executing an application function (AF), the AF providing media control for the media communication session;

determine one of the valid ICE candidates and one of the QoS flows associated with the one of the valid ICE candidates;

establish the media communication session with the second client device using the determined one of the valid ICE candidates;

determine an association between a QoS specification and the one of the QoS flows for the media communication session; and

invoke QoS provided by the first MNO using a service based architecture (SBA) procedure offered by the first MNO.

11. The first client device of claim 10 , wherein the second client device is communicatively coupled to a second MNO, and to determine the list of ICE candidates, the processing system is configured to receive data representing one or more of the ICE candidates from the first MNO.

12. The first client device of claim 10 , wherein the second client device is communicatively coupled to the first MNO.

13. The first client device of claim 10 , wherein the processing system is further configured to receive, from the server device executing the AF, data representative of at least one of a Traversal Using Relay Network Address Translation (TURN) server or a Session Traversal of UDP (STUN) server.

14. The first client device of claim 10 , wherein the data representing the QoS flows associated with the one or more of the valid ICE candidates includes data representing associations between QoS flow descriptions and QoS specifications.

15. The first client device of claim 10 , wherein the data representing the QoS flows associated with the one or more of the valid ICE candidates includes data describing a service data flow for which QoS is provided.

16. The first client device of claim 10 , wherein the data representing the QoS flows associated with the one or more of the valid ICE candidates includes a reference to the QoS specification.

17. The first client device of claim 10 , wherein the processing system is further configured to send an update message to the server device executing the AF including data associating one of the valid ICE candidates with one of the QoS flows.

18. The first client device of claim 10 , wherein the media communication session comprises a Web Real-Time Communication (WebRTC) communication session.

19. The first client device of claim 10 , further comprising a display.

20. The first client device of claim 10 , wherein the first client device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

21. A computer-readable storage medium having stored thereon instructions that, when executed, cause a processor of a first client device, communicatively coupled to a first mobile network operator (MNO), to:

determine a list of interactive connectivity establishment (ICE) candidates for a second client device;

determine valid ICE candidates in the list of ICE candidates;

for one or more of the valid ICE candidates, send data representing quality of service (QOS) flows associated with the one or more of the valid ICE candidates to a server device executing an application function (AF), the AF providing media control for the media communication session;

determine one of the valid ICE candidates and one of the QoS flows associated with the one of the valid ICE candidates;

establish the media communication session with the second client device using the determined one of the valid ICE candidates;

determine an association between a QoS specification and the one of the QoS flows for the media communication session; and

invoke QoS provided by the first MNO using a service based architecture (SBA) procedure offered by the first MNO.

22. The computer-readable storage medium of claim 21 , wherein the second client device is communicatively coupled to a second MNO, and the instructions that cause the processor to determine the list of ICE candidates comprise instructions that cause the processor to receive data representing one or more of the ICE candidates from the first MNO.

23. The computer-readable storage medium of claim 21 , wherein the second client device is communicatively coupled to the first MNO.

24. The computer-readable storage medium of claim 23 , further comprising instructions that cause the processor to receive, from the server device executing the AF, data representative of at least one of a Traversal Using Relay Network Address Translation (TURN) server or a Session Traversal of UDP (STUN) server.

25. The computer-readable storage medium of claim 23 , wherein the data representing the QoS flows associated with the one or more of the valid ICE candidates includes data representing associations between QoS flow descriptions and QoS specifications.

26. The computer-readable storage medium of claim 23 , wherein the data representing the QoS flows associated with the one or more of the valid ICE candidates includes data describing a service data flow for which QoS is provided.

27. The computer-readable storage medium of claim 23 , wherein the data representing the QoS flows associated with the one or more of the valid ICE candidates includes a reference to the QoS specification.

28. The computer-readable storage medium of claim 23 , further comprising instructions that cause the processor to send an update message to the server device executing the AF including data associating one of the valid ICE candidates with one of the QoS flows.

29. The computer-readable storage medium of claim 23 , wherein the media communication session comprises a Web Real-Time Communication (WebRTC) communication session.

30. A first client device for applying quality of service to a media communication session, the first client device being communicatively coupled to a first mobile network operator (MNO), the first client device comprising:

means for determining a list of interactive connectivity establishment (ICE) candidates for a second client device;

means for determining valid ICE candidates in the list of ICE candidates;

means for sending, for one or more of the valid ICE candidates, data representing quality of service (QOS) flows associated with the one or more of the valid ICE candidates to a server device executing an application function (AF), the AF providing media control for the media communication session;

means for determining one of the valid ICE candidates and one of the QoS flows associated with the one of the valid ICE candidates;

means for establishing the media communication session with the second client device using the determined one of the valid ICE candidates;

means for determining an association between a QoS specification and the one of the QoS flows for the media communication session; and

means for invoking QoS provided by the first MNO using a service based architecture (SBA) procedure offered by the first MNO.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 4, 2023
From: BOUAZIZI, IMED; STOCKHAMMER, THOMAS; LEUNG, NIKOLAI KONRAD
To: QUALCOMM INCORPORATED
Reel/Frame 065125/0293 →
Continuity (3)
Continuation 17818566 · Aug 9, 2022
Provisional Application 63232005 · Aug 11, 2021
Related Publication 20240121288A1 · Apr 11, 2024
References Cited (59)
US 11191013B1 · Kalkunte · 2021 [cited by examiner]
US 20160174239A1 · Tietsch et al. · 2016 [cited by applicant]
US 20200344084A1 · Shribman · 2020 [cited by examiner]
US 20220103525A1 · Shribman et al. · 2022 [cited by applicant]
US 20220287083A1 · Gomes Baltar · 2022 [cited by examiner]
US 20230064154A1 · Bouazizi et al. · 2023 [cited by applicant]
3GPP TR 21.905: “3rd Generation Partnership Project, Technical Specification Group Services and System Aspects, Vocabulary for 3GPP Specifications (Release 17)”, V17.1.0, Dec. 2021, pp. 1-65. [cited by applicant]
3GPP TR 22.873: “3rd Generation Partnership Project, Technical Specification Group Services and System Aspects, Study on Evolution of IMS Multimedia Telephony Service (Release 18)”, V18.0.0, Jun. 2021, pp. 1-18. [cited by applicant]
3GPP TR 23.701: “3rd Generation Partnership Project, Technical Specification Group Services and System Aspects, Study on Web Real Time Communication (WebRTC) access to IP Multimedia Subsystem (IMS), Stage 2 (Release 12)… [cited by applicant]
3GPP TR 23.706: “3rd Generation Partnership Project, Technical Specification Group Services and System Aspects, Study on Enhancements to Web Real Time Communication (WebRTC) Access to IP Multimedia Subsystem (IMS), Stag… [cited by applicant]
3GPP TR 26.926: “3rd Generation Partnership Project, Technical Specification Group Services and System Aspects, Traffic Models and Quality Evaluation Methods for Media and XR Services in 5G Systems, (Release 18)”, V1.3.… [cited by applicant]
3GPP TR 26.928: “3rd Generation Partnership Project, Technical Specification Group Services and System Aspects, Extended Reality (XR) in 5G (Release 17)”, V17.0.0, Apr. 2022, 130 pages. [cited by applicant]
3GPP TR 26.998: “3rd Generation Partnership Project, Technical Specification Group Services and System Aspects, Support of 5G Glass-type Augmented Reality / Mixed Reality (AR/MR) Devices, (Release 17)”, 3GPP TR 26.998 V… [cited by applicant]
3GPP TR 26.998: “3rd Generation Partnership Project, Technical Specification Group Services and System Aspects, Support of 5G Glass-type Augmented Reality / Mixed Reality (AR/MR) Devices, (Release 18)”, 3GPP TR 26.998 V… [cited by applicant]
3GPP TR 38.838: “3rd Generation Partnership Project, Technical Specification Group Radio Access Network, Study on XR (Extended Reality) Evaluations for NR (Release 17)”, V17.0.0, Dec. 2021, pp. 1-269. [cited by applicant]
3GPP TS 22.261: “3rd Generation Partnership Project, Technical Specification Group Services and System Aspects, Service Requirements for the 5G system, Stage 1 (Release 19)”, V19.1.0, Dec. 2022, 115 pages. [cited by applicant]
3GPP TS 26.261: “3rd Generation Partnership Project, Technical Specification Group Services and System Aspects, Terminal Audio Quality Performance Requirements for Immersive Audio Services Release 17”, V0.0.1, Apr. 2019… [cited by applicant]
3GPP Ts 26.512: “3rd Generation Partnership Project; Technical Specification Group Services and System Aspects; 5G Media Streaming (5GMS); Protocols (Release 16)”, 3GPP TS 26.512 V16.4.0 (Sep. 2021), pp. 1-135. [cited by applicant]
Bay., et al., “SURF: Speeded Up Robust Features” European Conference on Computer Vision, 2006, pp. 404-417. [cited by applicant]
Chen H., et al., “Understanding the Characteristics of Mobile Augmented Reality Applications”, 2018 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), Apr. 2018, pp. 128-138. [cited by applicant]
Daniel W., et al., “Why is Making Good AR Displays So Hard?”, LinkedIn Blog, Aug. 7, 2019, pp. 1-38. [cited by applicant]
Drage K., et al., “Negotiation Data Channels Using the Session Description Protocol (SDP)”, Internet Engineering Task Force (IETF), RFC: 8864, Category: Standards Track, Jan. 2021, pp. 1-24. [cited by applicant]
ETSI GS ARF 003, “Augmented Reality Framework (ARF), AR Framework Architecture”, European Telecommunications Standards Institute (ETSI), 650, Route Des Lucioles, F-06921 Sophia-Antipolis, France, V1.1.1, Mar. 2020, pp. … [cited by applicant]
ETSI TS 123 501: “5G, System Architecture for the 5G System (5GS)”, 3GPP TS 23.501 version 16.6.0 Release 16, Oct. 2020, pp. 1-449. [cited by applicant]
ETSI TS 124 371: “Universal Mobile Telecommunications System (UMTS), LTE, Web Real-Time Communications (WebRTC) Access to the IP Multimedia (IM) Core Network (CN) subsystem (IMS), Stage 3, Protocol specification”, 3GPP … [cited by applicant]
ETSI TS 126 260: “5G, Objective Test Methodologies for the Evaluation of Immersive Audio Systems (3GPP Ts 26.260 version 16.0.0 Release 16)”, V16.0.0, Aug. 2020, pp. 1-26. [cited by applicant]
ETSI TS 126 501: “5G, 5G Media Streaming (5GMS), General Description And Architecture (3GPP TS 26.501 version 16.5.0 Release 16)”, V16.5.0, Oct. 2020, 74 Pages. [cited by applicant]
ETSI TS 129 514: “5G System, Policy Authorization Service, Stage 3 (3GPP TS 29.514 version 16.6.0 Release 16)”, Nov. 2020, pp. 1-179. [cited by applicant]
Falmer O., “AR Headsets Landscape”, Sep. 10, 2021. [cited by applicant]
Google: “Google WebRTC Project Update & Stadia Review”, YouTube, Retrieved on Mar. 22, 2023, pp. 1-4. [cited by applicant]
Google: “Use Depth in Your Android App”, Google Developers, Last updated: May 20, 2022, pp. 1-10. [cited by applicant]
Gul S., et al., “Cloud Rendering-Based Volumetric Video Streaming System for Mixed Reality Services”, Proceedings of the 11th ACM Multimedia Systems Conference, May 2020, 4 pages. [cited by applicant]
Gunkel S.N.B., et al., “Virtual Reality Conferencing: Multi-User Immersive VR Experiences on the Web”, Proceedings of the 9th ACM Multimedia Systems Conference, Jun. 2018, 4 pages. [cited by applicant]
International Search Report and Written Opinion—PCT/US2022/074759—ISA/EPO—Nov. 18, 2022. [cited by applicant]
ISO: “Information Technology—Coded Representation of Immersive Media—Part 3: Versatile video coding”, ISO/IEC JTC 1/SC 29 /WG 11 N18692, ISO/IEC CD 23090-3, Jul. 12, 2019, 446 Pages. [cited by applicant]
ISO/IEC JTC 1/SC 29/WG11 [23090-2:201x (E) FDIS] “Information Technology—Coded Representation of Immersive Media (MPEG-I)—Part 2: Omnidirectional Media Format”, Apr. 26, 2018, 182 pages. [cited by applicant]
“ITU-T H.265, Series H: Audiovisual and Multimedia Systems, Infrastructure of Audiovisual Services—Coding of Moving Video, High Efficiency Video Coding”, The International Telecommunication Union, Apr. 2015, 634 Pages, … [cited by applicant]
ITU-T H.266: “Series H: Audiovisual and Multimedia Systems Infrastructure of Audiovisual Services—Coding of Moving Video”, Versatile Video Coding, The International Telecommunication Union, Aug. 2020, 516 pages. [cited by applicant]
Jesup R., et al., “WebRTC Data Channels”, Internet Engineering Task Force (IETF), RFC: 8831, Category: Standards Track, Jan. 2021, pp. 1-14. [cited by applicant]
Kang S., et al., “Fire in Your Hands: Understanding Thermal Behavior of Smartphones”, The 25th Annual International Conference on Mobile Computing and Networking, May 2019, 16 pages. [cited by applicant]
Lowe D.G., “Distinctive Image Features from Scale-Invariant Key Points,” International Journal of Computer Vision, vol. 60, No. 2, Jan. 5, 2004, XP055203065, 28 pages. [cited by applicant]
Microsoft: “Scene Lighting”, AZURE, Jun. 8, 2022, pp. 1-3. [cited by applicant]
Microsoft: “Scene Understanding”, Mixed Reality, Sep. 21, 2022, pp. 1-6. [cited by applicant]
Microsoft: “Color Materials”, Azure, Jan. 9, 2023, pp. 1-4. [cited by applicant]
Microsoft: “Coordinate Systems”, Mixed Reality, Jan. 18, 2023, pp. 1-11. [cited by applicant]
Microsoft: “PBR Materials”, AZURE, Jan. 9, 2023, pp. 1-9. [cited by applicant]
Microsoft: “Use Azure Kinect Sensor SDK Image Transformations”, Azure, Sep. 21, 2022, pp. 1-6. [cited by applicant]
MPEG131 Press Release, “MPEG131 Press Release: Point Cloud Compression—WG11(MPEG) Promotes a Video-based Point Cloud Compression Technology to the FDIS Stage”, Multimedia Communication, Jul. 16, 2020, pp. 1-2. [cited by applicant]
Oscar Falmer: “Mobile AR Features Landscape”, Last Update: Sep. 20, 2021, 4 pages. [cited by applicant]
Rescorla E., “WebRTC Security Architecture”, Internet Engineering Task Force (IETF), RFC: 8827, Category: Standards Track, Jan. 2021, pp. 1-35. [cited by applicant]
Rosenberg J., et al., “Session Traversal Utilities for NAT (STUN); rfc5389.txt”, Session Traversal Utilities for NAT (STUN); RFC5389.TXT, Internet Engineering Task Force, IETF, Standard, Internet Society (ISOC) 4, Rue D… [cited by applicant]
Rosenberg J., Interactive Connectivity Establishment (ICE): A Protocol for Network Address Translator (NAT) Traversal for Offer/Answer Protocols, Internet Engineering Task Force (IETF), RFC: 5245, Obsoletes: 4091, 4092,… [cited by applicant]
Rublee E., et al., “ORB: an Efficient Alternative to SIFT or SURF”, International Conference on Computer Vision, Nov. 2011, 9 pages. [cited by applicant]
Soudarissanane S.D., et al., “Multi-Sensor Capture and Network Processing for Virtual Reality Conferencing”, Proceedings of the 10th ACM Multimedia Systems Conference, Jun. 2019, 4 pages. [cited by applicant]
Takanori C., et al., “Evaluation of Physical Workload Affected by Mass and Center of Mass of Head-Mounted Display”, Applied Ergonomics, Apr. 2018, pp. 1-12. [cited by applicant]
W3C: “WebXR Device API”, W3C Candidate Recommendation Draft, Mar. 3, 2023, pp. 1-123. [cited by applicant]
Wagner D., “Motion to Photon Latency in Mobile AR and VR”, DAQRI, Aug. 20, 2018, pp. 1-15. [cited by applicant]
YADAYODA: “Why Loop Closure is so Important for Global Mapping”, Dec. 23, 2020, pp. 1-7. [cited by applicant]
Younes G., et al., “Keyframe-Based Monocular SLAM: Design, Survey, and Future Directions”, Robotics and Autonomous Systems, vol. 98, Dec. 2017, pp. 1-25. [cited by applicant]