IP Library Granted Patent US 11,398,020
Granted Patent B2
US 11,398,020 · App. 17/132,229 · Granted Jul 26, 2022

Generating a composite image

Inventors: Rohit Rao Padebettu (Cupertino, CA); Steven Paul Lansel (East Palo Alto, CA); Todd Douglas Keeler (Fremont, CA)
Assignee: Facebook Technologies, LLC.
G06T5/50G06T3/0093G06T5/006G06T2207/10016G06T2207/20004G06T2207/20221
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,398,020
App. No.
17/132,229
Granted
Jul 26, 2022
Kind
B2
Abstract

In one embodiment, a method includes receiving a source image and its associated parameters from each of multiple image sources, associating each of the source images with a layer in a range of layers based on the parameters associated with the source images, the range of layers specifying a composition layering order of the source images, generating a corresponding customized distortion mesh for each particular source image in the source images based on the parameters associated with the particular source image and at least a portion of the parameters associated with each of the source images that is associated with any layer preceding a layer associated with the particular source image, modifying each of the source images using the corresponding customized distortion mesh, generating a composite image using the modified source images, and displaying the composite image as a frame in a video.

Claims (42)

1. A method comprising, by a compositor module of a computing device:

receiving, from each of n image sources, a source image and its associated parameters, wherein the n image sources include at least two image sources;

associating each of the source images with a layer in a range of layers based on the parameters associated with the source images, the range of layers specifying a composition layering order of the source images;

generating, for each particular source image in the source images, a corresponding customized distortion mesh based on the parameters associated with the particular source image and at least a portion of the parameters associated with each of the source images that is associated with any layer preceding a layer associated with the particular source image;

modifying each of the source images using the corresponding customized distortion mesh;

generating a composite image using the modified source images; and

displaying the composite image as a frame in a video.

2. The method of claim 1 , wherein the parameters associated with each of the source images comprise a layer parameter, and wherein a layer is associated with the source image based on the layer parameter associated with the source image.

3. The method of claim 1 , wherein the parameters associated with each of the source images comprise clipping parameters indicating a bounding area for the source image, an area within the output image that is to be filled by the source image.

4. The method of claim 3 , wherein the clipping parameters associated with a source image associated with a last layer in the range of layers indicate that a whole area of the output image is the bounding area for the source image.

5. The method of claim 3 , wherein the parameters associated with the source image comprise reprojection parameters comprising information associated with one or more warping operations to be performed on the source image.

6. The method of claim 5 , wherein the bounding area for the source image is updated based on an expected impact of the one or more warping operations.

7. The method of claim 6 , wherein an updated bounding area for a first source image of the source images is further updated by subtracting, for each second image of the source images that is associated with any layer preceding a layer associated with the first source image, an intersection between the updated bounding area for the first source image and the updated bounding area for the second source image from the updated bounding area for the first source image.

8. The method of claim 7 , wherein the parameters associated with the source image comprise blending parameters comprising information associated with a transition area for the source image, wherein the transition area is a subject of an alpha blending.

9. The method of claim 8 , wherein the transition area for the source image is a subset of the updated bounding area for the source image, wherein the blending parameters comprise a cutoff distance, and wherein a point within the cutoff distance from a boundary of the updated bounding area for the source image belongs to the transition area for the source image.

10. The method of claim 8 , wherein the blending parameters associated with the source image comprise a level of transparency to be applied to the transition area for the source image.

11. The method of claim 8 , wherein generating the customized distortion mesh corresponding to each of the source images comprises generating a distortion mesh that matches to the updated bounding area for the source image.

12. The method of claim 11 , wherein modifying each of the source images comprises:

performing the one or more warping operations on the source image based on the associated reprojection parameters;

mapping the warped source image onto the corresponding customized distortion mesh; and

performing an alpha blending to the transition area for the source image.

13. The method of claim 5 , wherein the one or more warping operations comprise a space warping operation, wherein the reprojection parameters comprise motion vectors in the source image.

14. The method of claim 5 , wherein the one or more warping operations comprise a time warping operation, wherein the reprojection parameters comprise a head position.

15. The method of claim 3 , further comprising:

sending, to each of the n image sources, a composition result comprising parameter suggestions, wherein the image source adjusts parameters associated with a source image corresponding to a next frame based on the parameter suggestions.

16. The method of claim 15 , wherein the composition result comprises a result code indicating whether the bounding area for the source image is fully, partially or zero occluded by the bounding areas for the other source images among the source images.

17. The method of claim 16 , wherein the parameter suggestions comprise a new layer suggestion that is determined based on the result code.

18. The method of claim 1 , wherein the displayed composite image is viewed by a user through a lens systems, and wherein the customized distortion mesh corresponding to each of the source images is used to compensate a lens distortion caused by the lens system.

19. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

receive, from each of n image sources, a source image and its associated parameters, wherein the n image sources include at least two image sources;

associate each of the source images with a layer in a range of layers based on the parameters associated with the source images, the range of layers specifying a composition layering order of the source images;

generate, for each particular source image in the source images, a corresponding customized distortion mesh based on the parameters associated with the particular source image and at least a portion of the parameters associated with each of the source images that is associated with any layer preceding a layer associated with the particular source image;

modify each of the source images using the corresponding customized distortion mesh;

generate a composite image using the modified source images; and

display the composite image as a frame in a video.

20. A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:

receive, from each of n image sources, a source image and its associated parameters, wherein the n image sources include at least two image sources;

associate each of the source images with a layer in a range of layers based on the parameters associated with the source images, the range of layers specifying a composition layering order of the source images;

generate, for each particular source image in the source images, a corresponding customized distortion mesh based on the parameters associated with the particular source image and at least a portion of the parameters associated with each of the source images that is associated with any layer preceding a layer associated with the particular source image;

modify each of the source images using the corresponding customized distortion mesh;

generate a composite image using the modified source images; and

display the composite image as a frame in a video.

Assignments (2)
CHANGE OF NAME Recorded Jul 6, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060591/0848 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 6, 2021
From: RAO PADEBETTU, ROHIT; LANSEL, STEVEN PAUL; KEELER, TODD DOUGLAS
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 054827/0465 →