IP Library Granted Patent US 11,595,646
Granted Patent B2
US 11,595,646 · App. 16/912,290 · Granted Feb 28, 2023

Sliced encoding and decoding for remote rendering

Inventors: Matthew Robert Fulghum (San Mateo, CA); Reza Nourai (Danville, CA); Behnam Bastani (Palo Alto, CA)
Assignee: Meta Platforms Technologies, LLC
H04N19/119G02B27/0172G06F3/011G06F3/017G06T19/006H04L65/70H04L65/75H04N19/174
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,595,646
App. No.
16/912,290
Granted
Feb 28, 2023
Kind
B2
Abstract

Disclosed herein are related to a device and a method of remotely rendering an image. In one approach, a device divides an image of an artificial reality space into a plurality of slices. In one approach, the device encodes a first slice of the plurality of slices. In one approach, the device encodes a portion of a second slice of the plurality of slices, while the device encodes a portion of the first slice. In one approach, the device transmits the encoded first slice of the plurality of slices to a head wearable display. In one approach, the device transmits the encoded second slice of the plurality of slices to the head wearable display, while the device transmits a portion of the encoded first slice to the head wearable display.

Claims (55)

1. A device comprising:

a content generator comprising at least one processor, configured to:

partition an image of an artificial reality space into a plurality of slices,

encode a first slice of the plurality of slices during a first time period between a first time and a second time, the encoding of the first slice starting at the first time and completing at the second time, and

encode a second slice of the plurality of slices during a second time period between a third time and a fourth time, the encoding of the second slice starting at the third time and completing at the fourth time, wherein the third time is after the first time and before the second time; and

a communication interface coupled to the content generator, the communication interface configured to:

transmit the encoded first slice of the plurality of slices to a head wearable display during a third time period between a fifth time and a sixth time, wherein the fifth time is after the second time, and the transmission of the encoded first slice starts at the fifth time and completes at the sixth time, and

transmit the encoded second slice of the plurality of slices to the head wearable display during a fourth time period between a seventh time and an eighth time, wherein the seventh time is after the fourth time, and the transmission of the encoded second slice starts at the seventh time, and wherein the seventh time is after the fifth time and before the sixth time.

2. The device of claim 1 , wherein the first slice and the second slice are separated by a boundary.

3. The device of claim 2 , wherein the content generator is further configured to generate motion vectors of the image, wherein the motion vectors do not traverse the boundary between the first slice and the second slice.

4. The device of claim 1 , wherein the communication interface is configured to transmit another portion of the encoded first slice, while the content generator encodes a portion of the second slice.

5. The device of claim 1 ,

wherein the content generator is further configured to encode a third slice of the plurality of slices during a fifth time period between a ninth time and a tenth time, wherein the ninth time is after the third time and before the second time, and

wherein the communication interface is further configured to transmit the encoded third slice of the plurality of slices to the head wearable display during a sixth time period between an eleventh time and a twelfth time, wherein the eleventh time is after the seventh time and before the sixth time.

6. The device of claim 5 , wherein the communication interface is configured to transmit an additional portion of the encoded second slice, while the content generator encodes a portion of the third slice.

7. The device of claim 1 ,

wherein the communication interface is further configured to receive sensor measurements indicating a location or an orientation of the head wearable display, and

wherein the content generator is configured to generate the image of the artificial reality space according to the location or the orientation of the head wearable display.

8. A method comprising:

partitioning, by a device, an image of an artificial reality space into a plurality of slices;

encoding, by the device, a first slice of the plurality of slices during a first time period between a first time and a second time, the encoding of the first slice starting at the first time and completing at the second time;

encoding, by the device, a second slice of the plurality of slices during a second time period between a third time and a fourth time, the encoding of the second slice starting at the third time and completing at the fourth time, wherein the third time is after the first time and before the second time;

transmitting, by the device, the encoded first slice of the plurality of slices to a head wearable display during a third time period between a fifth time and a sixth time, wherein the fifth time is after the second time, and the transmission of the encoded first slice starts at the fifth time and completes at the sixth time; and

transmitting, by the device, the encoded second slice of the plurality of slices to the head wearable display during a fourth time period between a seventh time and an eighth time, wherein the seventh time is after the fourth time, and the transmission of the encoded second slice starts at the seventh time, and wherein the seventh time is after the fifth time and before the sixth time.

9. The method of claim 8 , wherein the first slice and the second slice are separated by a boundary.

10. The method of claim 9 , further comprising:

generating, by the device, motion vectors of the image, wherein the motion vectors do not traverse the boundary between the first slice and the second slice.

11. The method of claim 8 , further comprising:

transmitting, by the device, another portion of the encoded first slice, while the device encodes a portion of the second slice.

12. The method of claim 8 , further comprising:

encoding, by the device, a third slice of the plurality of slices during a fifth time period between a ninth time and a tenth time, wherein the ninth time is after the third time and before the second time; and

transmitting, by the device, the encoded third slice of the plurality of slices to the head wearable display during a sixth time period between an eleventh time and a twelfth time, wherein the eleventh time is after the seventh time and before the sixth time.

13. The method of claim 12 , further comprising:

transmitting, by the device, an additional portion of the encoded second slice, while the device encodes a portion of the third slice.

14. The method of claim 8 , further comprising:

receiving, by the device, sensor measurements indicating a location or an orientation of the head wearable display; and

generating, by the device, the image of the artificial reality space according to the location or the orientation of the head wearable display.

15. A device comprising:

a communication interface configured to:

receive, from another device, an encoded first slice of an image of an artificial reality space during a first time period between a first time and a second time, the encoding of the first slice starting at the first time and completing at the second time, and

receive, from the another device, an encoded second slice of the image during a second time period between a third time and a fourth time, the encoding of the second slice starting at the third time and completing at the fourth time, wherein the third time is after the first time and before the second time; and

an image renderer comprising at least one processor, the image renderer coupled to the communication interface, the image renderer configured to:

decode the encoded first slice of the image during a third time period between a fifth time and a sixth time, wherein the fifth time is after the second time, and the decoding of the encoded first slice starts at the fifth time and completes at the sixth time,

decode the encoded second slice of the image during a fourth time period between a seventh time and an eighth time, wherein the seventh time is after the fourth time, and the decoding of the encoded second slice starts at the seventh time, and wherein the seventh time is after the fifth time and before the sixth time,

combine the decoded first slice of the image and the decoded second slice of the image, and

render the image based on the combination of the decoded first slice of the image and the decoded second slice of the image.

16. The device of claim 15 , wherein the communication interface is configured to receive a portion of the encoded second slice, while the image renderer decodes an additional portion of the encoded first slice.

17. The device of claim 16 , wherein the decoded first slice and the decoded second slice are separated by a boundary.

18. The device of claim 17 , wherein motion vectors of the decoded first slice and motion vectors of the decoded second slice do not traverse the boundary between the decoded first slice and the decoded second slice.

19. The device of claim 15 ,

wherein the communication interface is further configured to receive an encoded third slice of the plurality of slices from the another device during a fifth time period between at a ninth time and a tenth time, wherein the ninth time is after the third time and before the second time, and

wherein the image renderer is further configured to decode the encoded third slice of the plurality of slices during a sixth time period between an eleventh time and a twelfth time, wherein the eleventh time is after the seventh time and before the sixth time.

20. The device of claim 15 , further comprising:

sensors configured to generate sensor measurements indicating a location or an orientation of the device,

wherein the communication interface is configured to transmit the sensor measurements to the another device, and receive the encoded first slice and the encoded second slice of the image, in response to transmitting the sensor measurements.

Assignments (2)
CHANGE OF NAME Recorded Jul 22, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060816/0634 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 16, 2020
From: FULGHUM, MATTHEW ROBERT, DR.; NOURAI, REZA; BASTANI, BEHNAM
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 053234/0482 →