Method and apparatus for combining an augmented reality object in a real-world image

1 . A method comprising:

receiving a first depth image associated with a first frame of an augmented reality (AR) application, the first depth image including depth layers and the first depth image representing at least a first portion of a real-world space;

generating a blended depth image by,

identifying missing or invalid depth information in a first depth layer of the first depth image,

identifying a second depth image from a buffer associated with the AR application, the second depth image including a second depth layer corresponding to the first depth layer and including the missing or invalid depth information and the buffer stores depth images associated with frames captured before the first frame,

projecting the second depth image into a perspective associated with the first depth image, and

using the first depth image and depth information from the second depth image that has been projected and corresponds to the missing or invalid depth information of the second depth layer to generate the blended depth image; and

displaying an AR object combined with a real-world image generated based on the blended depth image.

2 . The method of claim 1 , wherein the second depth image is one of a plurality of depth images representing frames of the AR application stored in the buffer.

3 . The method of claim 1 , wherein the second depth image is one of a plurality of depth images representing frames of the AR application stored in the buffer, the method further comprising:

accumulating the plurality of depth images including projecting at least one stored frame into a current frame;

selecting one of the plurality of depth images stored in the buffer as the second depth image;

generating a data structure based on the second depth image, the data structure representing the real-world space, the data structure including depth information, position information, and direction information; and

storing the data structure.

4 . The method of claim 1 , wherein the second depth image is one of a plurality of depth images representing frames of the AR application stored in the buffer, the method further comprising:

receiving a portion of the plurality of depth images stored in the buffer;

generating a plurality of surface elements (surfels) based on the portion of the plurality of depth images, the plurality of surfels representing the real-world space; and

storing the plurality of surfels.

5 . The method of claim 1 , further comprising:

receiving a data structure including depth information, position information, and direction information, the depth information including the depth layers; and

rendering the data structure as the second depth image including aligning the direction information based on a direction associated with the first depth image, wherein generating the blended depth image includes:

identifying a depth layer of the first depth image that includes the missing or invalid depth information as the first depth layer, and

generating the blended depth image includes using the data structure to identify a corresponding depth layer of the second depth image as the second depth layer, and

replacing a portion of the first depth layer to generate the blended depth image; and

blending the blended depth image with the real-world image.

6 . The method of claim 1 , further comprising:

receiving a plurality of surfels representing the real-world space;

rendering the plurality of surfels as a third depth image; and

blending the third depth image with the real-world image.

7 . The method of claim 1 , further comprising:

combining the real-world image with the AR object by replacing a portion of pixels in the real-world image with a portion of pixels in the AR object based on depth.

8 . The method of claim 1 , wherein the generating of the blended depth image includes replacing a portion of pixels in the first depth image with a portion of the second depth image.

9 . The method of claim 1 , wherein

the missing or invalid depth information corresponds to at least one pixel; and

the generating of the blended depth image includes replacing the at least one pixel with a portion of the first depth image.

10 . The method of claim 1 , further comprising:

receiving a plurality of surfels representing the real-world space; and

rendering the plurality of surfels, wherein

the first depth image is missing at least one pixel, and

the generating of the blended depth image includes replacing the at least one pixel with a portion of the plurality of surfels.

11 . The method of claim 1 , wherein the first depth image includes a position confidence indicating a likelihood that the second depth image represents the real-world space at a position.

12 . A method comprising:

receiving a depth data associated with a frame of an augmented reality (AR) application, the depth data representing at least a portion of a real-world space, the depth data including depth layers and a perspective;

storing the depth data as one of a plurality of depth images representing frames of the AR application in a buffer associated with the AR application;

identifying missing or invalid depth information in a depth layer of the depth data;

selecting a portion of the plurality of depth images corresponding to depth layer including the missing or invalid depth information from the plurality of depth images stored in the buffer, the plurality of depth images associated with frames captured before the frame;

projecting the portion of the plurality of depth images into the perspective associated with the depth data by replacing the missing or invalid depth information in the depth layer of the depth data;

generating a data structure based on the depth data and the projected portion of the plurality of depth images, the depth data being received later in time than the portion of the plurality of depth images, the data structure representing the real-world space, the data structure including depth information, position information, and direction information; and

storing the generated data structure.

13 . The method of claim 12 , wherein the selecting of the portion of the plurality of depth images includes selecting the plurality of images from a plurality of buffers on a plurality of devices executing the AR application.

14 . The method of claim 12 , wherein the plurality of depth images include a position confidence indicating a likelihood that the depth data represents the real-world space at a position.

15 . A method comprising:

receiving a depth data associated with a frame of an augmented reality (AR) application, the depth data including first depth layers and the depth data representing at least a first portion of a real-world space;

receiving a data structure representing at least a second portion of the real-world space associated with the AR application, the data structure including second depth layers including depth information, position information, and direction information, the data structure being generated by accumulating a plurality of depth images including projecting at least one stored frame into a current frame based on a direction associated with the depth data;

generating a blended depth image by,

identifying missing or invalid depth information in a depth layer of the depth data, and

using the depth information from a corresponding depth layer of the data structure that corresponds to the missing or invalid depth information to generate the blended depth image the depth data being received later in time than the depth information associated with the data structure; and

displaying an AR object combined with a real-world image generated based on the blended depth image.

16 . The method of claim 15 , further comprising:

combining the real-world image with the AR object by replacing a portion of pixels in the real-world image with a portion of pixels in the AR object based on depth.

17 . The method of claim 15 , wherein the generating of the blended depth image includes replacing a portion of pixels in the depth data with a portion of the data structure.

18 . The method of claim 15 , wherein

the depth data is missing at least one pixel; and

the generating of the blended depth image includes replacing the at least one pixel with a portion of the data structure.

19 . The method of claim 15 , wherein

the data structure includes a plurality of surfels, and

the depth data is missing at least one pixel, the method further comprising replacing the at least one pixel with a portion of the plurality of surfels.

20 . The method of claim 15 , wherein the data structure representing the real-world space includes a position confidence indicating a likelihood that the depth data represents the real-world space at a position.