Granted Patent B2

US 12,277,264 · App. 18/037,709 · Granted Apr 15, 2025

Inventor: Praveen Babu J D (Plantation, FL)

Assignee: Magic Leap, Inc.

G06F3/013G06T7/50G06V10/25H04N19/59

Abstract

Claims (55)

1. A computer-implemented method comprising:

receiving gaze information about an observer of a video stream, wherein the gaze information includes information about instantaneous eye position and instantaneous eye velocity;

determining a video compression spatial map for the video stream based on the received gaze information and performance characteristics of a network connection with the observer, wherein the determining of the video compression spatial map includes:

selecting a first shape for a region of interest;

selecting a first video compression profile that includes higher compression outside the first shape;

selecting a second shape within the first shape of the region of interest and sharing a center of the region of interest;

scaling a size of the second shape in proportion to a network latency times a maximum eye velocity; and

selecting a second video compression profile that includes: (1) lower compression inside the second shape; (2) medium compression outside the second shape, but inside the first shape; and (3) higher compression outside the first shape, and

wherein the performance characteristics of the network connection include information about the network latency, and comprising:

identifying the center of the region of interest corresponding to a predicted eye position, wherein the center of the region of interest corresponds to the instantaneous eye position plus an offset proportional to the instantaneous eye velocity times the network latency;

compressing the video stream according to the video compression spatial map; and

sending the compressed video stream to the observer.

2. The method of claim 1 , wherein the receiving of the gaze information is a receiving of gaze information from a head-mounted or display-mounted gaze tracker.

3. The method of claim 1 , wherein:

the first video compression profile corresponds to a video quality profile, a video resolution profile, or a video color profile.

4. The method of claim 1 , wherein the first video compression profile increases in compression with distance from the center of the region of interest.

5. The method of claim 1 , wherein:

the performance characteristics of the network connection include information about available bandwidth; and

the determining of the video compression spatial map includes scaling a size of the first shape in proportion to a ratio of the available bandwidth to a bandwidth for the video stream without compression.

6. The method of claim 1 , wherein the lower compression is zero compression.

7. The method of claim 1 , wherein the maximum eye velocity is a human saccadic eye velocity.

8. A computer-implemented system, comprising:

one or more processors; and

one or more computer memory devices interoperably coupled with the one or more processors and having tangible, non-transitory, machine-readable media storing one or more instructions that, when executed by the one or more processors, perform one or more operations, comprising:

receiving gaze information about an observer of a video stream, wherein the gaze information includes information about instantaneous eye position and instantaneous eye velocity;

determining a video compression spatial map for the video stream based on the received gaze information and performance characteristics of a network connection with the observer, wherein the determining of the video compression spatial map includes:

selecting a first shape for a region of interest;

selecting a first video compression profile that includes higher compression outside the first shape;

selecting a second shape within the first shape of the region of interest and sharing a center of the region of interest;

scaling a size of the second shape in proportion to a network latency times a maximum eye velocity; and

selecting a second video compression profile that includes: (1) lower compression inside the second shape; (2) medium compression outside the second shape, but inside the first shape; and (3) higher compression outside the first shape, and

wherein the performance characteristics of the network connection include information about the network latency, and comprising:

identifying the center of the region of interest corresponding to a predicted eye position, wherein the center of the region of interest corresponds to the instantaneous eye position plus an offset proportional to the instantaneous eye velocity times the network latency;

compressing the video stream according to the video compression spatial map; and

sending the compressed video stream to the observer.

9. The computer-implemented system of claim 8 , wherein the receiving of the gaze information is a receiving of gaze information from a head-mounted or display-mounted gaze tracker.

10. The computer-implemented system of claim 8 , wherein:

the first video compression profile corresponds to a video quality profile, a video resolution profile, or a video color profile.

11. The computer-implemented system of claim 8 , wherein the first video compression profile increases in compression with distance from the center of the region of interest.

12. A non-transitory, computer-readable medium storing one or more instructions executable by a computer system to perform one or more operations, comprising:

receiving gaze information about an observer of a video stream, wherein the gaze information includes information about instantaneous eye position and instantaneous eye velocity;

determining a video compression spatial map for the video stream based on the received gaze information and performance characteristics of a network connection with the observer, wherein the determining of the video compression spatial map includes:

selecting a first shape for a region of interest;

selecting a first video compression profile that includes higher compression outside the first shape;

selecting a second shape within the first shape of the region of interest and sharing a center of the region of interest;

scaling a size of the second shape in proportion to a network latency times a maximum eye velocity; and

selecting a second video compression profile that includes: (1) lower compression inside the second shape; (2) medium compression outside the second shape, but inside the first shape; and (3) higher compression outside the first shape, and

wherein the performance characteristics of the network connection include information about the network latency, and comprising:

identifying the center of the region of interest corresponding to a predicted eye position, wherein the center of the region of interest corresponds to the instantaneous eye position plus an offset proportional to the instantaneous eye velocity times the network latency;

compressing the video stream according to the video compression spatial map; and

sending the compressed video stream to the observer.

13. The non-transitory, computer-readable medium of claim 12 , wherein the receiving of the gaze information is a receiving of gaze information from a head-mounted or display-mounted gaze tracker.

14. The non-transitory, computer-readable medium of claim 12 , wherein:

the first video compression profile corresponds to a video quality profile, a video resolution profile, or a video color profile.

15. The non-transitory, computer-readable medium of claim 12 , wherein the first video compression profile increases in compression with distance from the center of the region of interest.

Assignments (3)

Continuity (2)

References Cited (15)