IP Library Granted Patent US 11,881,192
Granted Patent B2
US 11,881,192 · App. 17/543,208 · Granted Jan 23, 2024

Compensating for latency in a streaming virtual reality environment

Inventors: Dhananjay Lal (Englewood, CO); Andrew Ip (Centennial, CO); Matthew Ringenberg (Denver, CO); Dell Wolfensparger (Langley, WA); Mun G. Choi (Denver, CO)
Assignee: Charter Communications Operating, LLC
G09G5/37G06F3/012G09G5/14H04L43/0864G09G2354/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,881,192
App. No.
17/543,208
Filed
Dec 6, 2021
Granted
Jan 23, 2024
Kind
B2
Art Unit
2629
USPC
345/156
Abstract

Compensating for latency in a streaming virtual reality environment is disclosed. A computing device receives from a head-mounted display device (HMDD) having a field of view (FOV), first pose data that identifies an orientation of the HMDD worn by a participant viewing a virtual world via the HMDD. The computing device renders, based at least in part on the first pose data and a first oversized FOV that is greater than the FOV, oversized imagery that depicts a portion of the virtual world within the FOV of the HMDD and a portion of the virtual world outside of the FOV of the HMDD. The computing device communicates the oversized imagery to the HMDD.

Claims (72)

1. A method comprising:

determining, by a head-mounted display device (HMDD) having a field of view (FOV), first pose data that identifies an orientation of the HMDD worn by a participant viewing a virtual world via the HMDD;

sending, by the HMDD to a computing device via a network, the first pose data;

receiving, by the HMDD from the computing device, first oversized imagery generated based on the first pose data and a first oversized FOV that is larger than the FOV, wherein the first oversized imagery depicts a first oversized scene in the virtual world that is larger than a first scene encompassed by the FOV of the HMDD and the first oversized FOV is based at least in part on a predetermined head movement quantifier quantifying a rotational velocity of a head movement for the virtual world;

determining, by the HMDD, a real-time orientation of the HMDD;

extracting, from the first oversized imagery, an extracted portion of the first oversized imagery that depicts a portion of the virtual world encompassed by the FOV of the HMDD based on the real-time orientation of the HMDD; and

presenting the extracted portion on a display device of the HMDD.

2. The method of claim 1 further comprising:

presenting, by the HMDD on the display device, a menu identifying a plurality of virtual worlds, including the virtual world;

receiving, by the HMDD, a selection of the virtual world; and

sending, to the computing device, information that identifies the FOV of the HMDD.

3. The method of claim 2 wherein sending the information that identifies the FOV of the HMDD further comprises sending information that identifies a horizontal FOV of the HMDD and a vertical FOV of the HMDD.

4. The method of claim 1 further comprising:

receiving, from the computing device, an internet control message protocol (ICMP) echo request; and

sending, by the HMDD to the computing device, a response to the ICMP echo request.

5. The method of claim 4 further comprising, subsequent to sending the response to the ICMP echo request:

determining, by the HMDD, second pose data that identifies the orientation of the HMDD worn by the participant viewing the virtual world via the HMDD;

sending, by the HMDD to the computing device via the network, the second pose data;

receiving, by the HMDD from the computing device, second oversized imagery generated based on the second pose data and a second oversized FOV that is larger than the FOV and smaller than the first oversized FOV, wherein the second oversized imagery depicts a second oversized scene in the virtual world that is larger than a second scene encompassed by the FOV of the HMDD;

determining, by the HMDD, the real-time orientation of the HMDD;

extracting, from the second oversized imagery, an extracted portion of the second oversized imagery that depicts a portion of the virtual world encompassed by the FOV of the HMDD based on the real-time orientation of the HMDD; and

presenting the extracted portion on the display device of the HMDD.

6. The method of claim 1 wherein the first oversized imagery comprises rendered imagery, and wherein the HMDD lacks a graphics processing unit (GPU) and the extracted portion of the first oversized imagery is extracted without rendering the extracted portion.

7. The method of claim 1 , wherein the first oversized FOV is based at least in part on a round-trip time value quantifying an elapsed time of a signal communicated from the computing device through the network to the HMDD and receipt, by the computing device, of a response from the HMDD to the signal.

8. A head mounted display device (HMDD), comprising:

a memory; and

a processor device coupled to the memory configured to:

determine first pose data that identifies an orientation of the HMDD worn by a participant viewing a virtual world via the HMDD;

send, to a computing device via a network, the first pose data;

receive, from the computing device, first oversized imagery generated based on the first pose data and a first oversized field of view (FOV) that is larger than an FOV of the HMDD, wherein the first oversized imagery depicts a first oversized scene in the virtual world that is larger than a first scene encompassed by the FOV of the HMDD and the first oversized FOV is based at least in part on a predetermined head movement quantifier quantifying a rotational velocity of a head movement for the virtual world;

determine a real-time orientation of the HMDD;

extract, from the first oversized imagery, an extracted portion of the first oversized imagery that depicts a portion of the virtual world encompassed by the FOV of the HMDD based on the real-time orientation of the HMDD; and

present the extracted portion on a display device of the HMDD.

9. The HMDD of claim 8 , wherein the processor device is further configured to:

present, on the display device, a menu identifying a plurality of virtual worlds, including the virtual world;

receive a selection of the virtual world; and

send, to the computing device, information that identifies the FOV of the HMDD.

10. The HMDD of claim 9 , wherein, to send the information that identifies the FOV of the HMDD, the processor device is further configured to send information that identifies a horizontal FOV of the HMDD and a vertical FOV of the HMDD.

11. The HMDD of claim 8 , wherein the processor device is further configured to:

receive, from the computing device, an internet control message protocol (ICMP) echo request; and

send, to the computing device, a response to the ICMP echo request.

12. The HMDD of claim 11 wherein, subsequent to sending the response to the ICMP echo request, the processor device is further configured to:

determine, second pose data that identifies the orientation of the HMDD worn by the participant viewing the virtual world via the HMDD;

send, to the computing device via the network, the second pose data;

receive, from the computing device, second oversized imagery generated based on the second pose data and a second oversized FOV that is larger than the FOV and smaller than the first oversized FOV, wherein the second oversized imagery depicts a second oversized scene in the virtual world that is larger than a second scene encompassed by the FOV of the HMDD;

determine the real-time orientation of the HMDD;

extract, from the second oversized imagery, an extracted portion of the second oversized imagery that depicts a portion of the virtual world encompassed by the FOV of the HMDD based on the real-time orientation of the HMDD; and

present the extracted portion on the display device of the HMDD.

13. The HMDD of claim 8 , wherein the first oversized imagery comprises rendered imagery, and wherein the HMDD lacks a graphics processing unit (GPU) and the extracted portion of the first oversized imagery is extracted without rendering the extracted portion.

14. A non-transitory computer-readable storage medium that includes executable instructions configured to cause a processor device to:

determine first pose data that identifies an orientation of a head-mounted display device (HMDD) having a field of view (FOV) worn by a participant viewing a virtual world via the HMDD;

send, to a computing device via a network, the first pose data;

receive, from the computing device, first oversized imagery generated based on the first pose data and a first oversized field of view (FOV) that is larger than an FOV of the HMDD, wherein the first oversized imagery depicts a first oversized scene in the virtual world that is larger than a first scene encompassed by the FOV of the HMDD and the first oversized FOV is based at least in part on a predetermined head movement quantifier quantifying a rotational velocity of a head movement for the virtual world;

determine a real-time orientation of the HMDD;

extract, from the first oversized imagery, an extracted portion of the first oversized imagery that depicts a portion of the virtual world encompassed by the FOV of the HMDD based on the real-time orientation of the HMDD; and

present the extracted portion on a display device of the HMDD.

15. The non-transitory computer-readable storage medium of claim 14 , wherein the instructions further cause the processor device to:

present, on the display device, a menu identifying a plurality of virtual worlds, including the virtual world;

receive a selection of the virtual world; and

send, to the computing device, information that identifies the FOV of the HMDD.

16. The non-transitory computer-readable storage medium of claim 15 , wherein, to send the information that identifies the FOV of the HMDD, the instructions further cause the processor device to send information that identifies a horizontal FOV of the HMDD and a vertical FOV of the HMDD.

17. The non-transitory computer-readable storage medium of claim 14 , wherein the instructions further cause the processor device to:

receive, from the computing device, an internet control message protocol (ICMP) echo request; and

send, to the computing device, a response to the ICMP echo request.

18. The non-transitory computer-readable storage medium of claim 17 wherein, subsequent to sending the response to the ICMP echo request, the instructions further cause the processor device to:

determine, second pose data that identifies the orientation of the HMDD worn by the participant viewing the virtual world via the HMDD;

send, to the computing device via the network, the second pose data;

receive, from the computing device, second oversized imagery generated based on the second pose data and a second oversized FOV that is larger than the FOV and smaller than the first oversized FOV, wherein the second oversized imagery depicts a second oversized scene in the virtual world that is larger than a second scene encompassed by the FOV of the HMDD;

determine the real-time orientation of the HMDD;

extract, from the second oversized imagery, an extracted portion of the second oversized imagery that depicts a portion of the virtual world encompassed by the FOV of the HMDD based on the real-time orientation of the HMDD; and

present the extracted portion on the display device of the HMDD.

19. The non-transitory computer-readable storage medium of claim 14 , wherein the first oversized imagery comprises rendered imagery, and wherein the HMDD lacks a graphics processing unit (GPU) and the extracted portion of the first oversized imagery is extracted without rendering the extracted portion.

Assignments (4)
SECURITY INTEREST Recorded Sep 22, 2022
From: CHARTER COMMUNICATIONS OPERATING, LLC; TIME WARNER CABLE ENTERPRISES, LLC
To: WELLS FARGO TRUST COMPANY, N.A.
Reel/Frame 061503/0937 →
SECURITY INTEREST Recorded Sep 22, 2022
From: CHARTER COMMUNICATIONS OPERATING, LLC; TIME WARNER CABLE ENTERPRISES, LLC
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A.
Reel/Frame 061504/0307 →
SUPPLEMENTAL SECURITY AGREEMENT Recorded Aug 10, 2022
From: CHARTER COMMUNICATIONS OPERATING, LLC; TIME WARNER CABLE ENTERPRISES LLC
To: BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 061633/0069 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 6, 2021
From: LAL, DHANANJAY; IP, ANDREW; RINGENBERG, MATTHEW; WOLFENSPARGER, DELL; CHOI, MUN G.
To: CHARTER COMMUNICATIONS OPERATING, LLC
Reel/Frame 058310/0360 →