Dynamic augmented reality overlay display
Three-dimensional occlusion can be used when generating AR display overlays. Depth information can be used to delete portions of an AR element, based on intervening objects between a viewer and the AR element. In cases where the depth information does not impart a complete picture of the intervening objects, additional image processing and object detection systems and techniques can be used to further improve the precision of the occlusion.
1. A method comprising:
receiving, by one or more processors of a computing device, a video stream of an environment, the video stream including a plurality of image frames, each respective image frame from the plurality of image frames including a projection of an environment point into one or more pixels of the respective image frame;
generating, by the one or more processors, a depth map of the image frame, the depth map including respective comprising distances of environment points to a user for each of the image pixels of the image frame;
determining, by the one or more processors, icon location of a virtual reality icon in the environment, the icon including a plurality of icon points;
determining, by the one or more processors, icon pixel coordinates of the virtual reality icon in the image frame based on a projection of the icon points into icon pixels in the image frame;
determining, by the one or more processors, icon distances of the plurality of the icon points from the user;
detecting, by the one or more processors, potential presence of an intervening object in the environment between the user and the icon location, based at least in part on the icon pixel coordinates, icon distances and the depth map;
performing, using an image segmentation module executed by the one or more processors, image segmentation on an image frame from the plurality of image frames;
labeling, by the one or more processors, each of the image pixels of the image frame with a corresponding segment label;
identifying, by the one or more processors, image pixels of the image frame having same corresponding segment labels as same-segment-label pixels;
determining, by the one or more processors, intervening object pixels, based at least in part on the depth map and the same-segment-label pixels overlapping the icon pixels and having pixel distances less than the icon distances;
determining, by the one or more processors, icon pixels overlapping the intervening object pixels;
generating, by the one or more processors, the icon with the overlapping icon pixels erased;
generating, by the one or more processors, an output image frame based on the image frame, the icon, and the overlapping same-segment-label pixels; and
outputting, by the one or more processors and for display, the output image frame.
2. The method of claim 1 , wherein detecting an intervening object comprises:
determining whether the icon pixel coordinates are in an area of the image having pixel distances closer to the user than the icon distances.
3. The method of claim 1 , wherein determining intervening object pixels comprise:
performing object detection on the image frame, tagging image pixels within an object box with an object tag;
determining the same-segment-label pixels having same object tags as same-segment label, same-object-tag image pixels; and
detecting same-segment-label, same-object-tag image pixels having pixel distances less than the icon distances as the intervening object pixels.
4. The method of claim 1 , further comprising:
generating a plurality of output image frames; and
generating an output video stream from the plurality of output image frames.
5. The method of claim 1 , wherein the plurality of image frames is a first plurality of image frames, the method further comprising:
generating an intervening object mask based on the intervening object pixels;
erasing icon pixels within a boundary of the intervening object mask;
generating the output image frame based on the image frame and the generated icon placed in the image frame at the icon pixel coordinates;
receiving a second plurality of image frames;
updating the intervening object mask between each image frame from the second plurality of image frames based on object and depth information in each image frame from the second plurality of image frames;
generating a plurality of output image frames, at least in part by generating icons in each image frame based on the intervening object masks; and
generating an output video stream from the plurality of the output image frames.
6. A non-transitory computer storage that stores executable program instructions that, when executed by one or more computing devices, configure the one or more computing devices to:
receive a video stream of an environment, the video stream including a plurality of image frames, each image frame comprising a projection of an environment point into one or more pixels in the image frame;
generate a depth map of the image frame, the depth map including respective distances of environment points to a user for each of the image pixels of the image frame;
determine icon location of a virtual reality icon in the environment, the icon including a plurality of icon points;
determine icon pixel coordinates of the virtual reality icon in the image frame, based on a projection of the icon points into icon pixels in the image frame;
determine icon distances of the plurality of the icon points from the user;
detect potential presence of an intervening object in the environment between the user and the icon location, based at least in part on the icon pixel coordinates, icon distances and the depth map;
perform image segmentation on an image frame from the plurality of image frames;
label each of the image pixels of the image frame with a corresponding segment label;
identify image pixels of the image frame having same corresponding segment labels as same-segment-label pixels;
determine intervening object pixels, based at least in part on the depth map and the same-segment-label pixels overlapping the icon pixels and having pixel distances less than the icon distances;
determine icon pixels overlapping the intervening object pixels;
generate the icon with the overlapping icon pixels erased generate an output image frame based on the image frame, the icon, and the overlapping same-segment-label pixels; and
output the output image frame.
7. The non-transitory computer storage of claim 6 , wherein the instructions that configure the one or more computing devices to detect an intervening object comprise instructions that cause the one or more computing devices to:
determine whether the icon pixel coordinates are in an area of the image having pixel distances closer to the user than the icon distances.
8. The non-transitory computer storage of claim 6 , wherein the instructions that configure the one or more computing devices to determine intervening object pixels comprise instructions that cause the one or more computing devices to:
perform object detection on the image frame, tagging image pixels within an object box with an object tag;
determine the same-segment-label pixels having same object tags as same-segment label, same-object-tag image pixels; and
detect same-segment-label, same-object-tag image pixels having pixel distances less than the icon distances as the intervening object pixels.
9. The non-transitory computer storage of claim 6 , wherein the instructions further cause the one or more computing devices to:
generate a plurality of output image frames; and
generate an output video stream from the plurality of output image frames.
10. The non-transitory computer storage of claim 6 , wherein the instructions further cause the one or more computing devices to:
generate an intervening object mask based on the intervening object pixels;
erase icon pixels within a boundary of the mask;
generate the output image frame based on the image frame and the generated icon placed in the image frame at the icon pixel coordinates;
receive a second plurality of image frames;
update the intervening object mask between each image frame from the second plurality of image frames based on object and depth information in each image frame from the second plurality of image frames;
generate a plurality of output image frames, at least in part by generating icons in each image frame based on the intervening object masks; and
generate an output video stream from the plurality of the output image frames.
11. A system comprising:
a processor; and
a storage device configured to store instructions that, when executed by the processor, cause the processor to:
receive a video stream of an environment, the video stream including a plurality of image frames, each image frame comprising a projection of an environment point into one or more pixels in the image frame;
generate a depth map of the image frame, the depth map including respective distances of environment points to a user for each of the image pixels of the image frame;
determine icon location of a virtual reality icon in the environment, the icon including a plurality of icon points;
determine icon pixel coordinates of the virtual reality icon in the image frame, based on a projection of the icon points into icon pixels in the image frame;
determine icon distances of the plurality of the icon points from the user;
detect potential presence of an intervening object in the environment between the user and the icon location, based at least in part on the icon pixel coordinates, icon distances and the depth map;
perform image segmentation on an image frame from the plurality of image frames;
label each of the image pixels of the image frame with a corresponding segment label;
identify image pixels of the image frame having same corresponding segment labels as same-segment-label pixels;
determine intervening object pixels, based at least in part on the depth map and the same-segment-label pixels overlapping the icon pixels and having pixel distances less than the icon distances;
determine icon pixels overlapping the intervening object pixels;
generate the icon with the overlapping icon pixels erased
generate an output image frame based on the image frame, the icon, and the overlapping same-segment-label pixels; and
output the output image frame.
12. The system of claim 11 , wherein the instructions that cause the processor to detect an intervening object comprise instructions that cause the processor to:
determine whether the icon pixel coordinates are in an area of the image having pixel distances closer to the user than the icon distances.
13. The system of claim 11 , wherein the instructions that cause the processor to determine intervening object pixels comprise instructions that cause the processor to:
perform object detection on the image frame, tagging image pixels within an object box with an object tag;
determine the same-segment-label pixels having same object tags as same-segment label, same-object-tag image pixels; and
detect same-segment-label, same-object-tag image pixels having pixel distances less than the icon distances as the intervening object pixels.
14. The system of claim 11 , wherein the instructions further cause the processor to:
generate a plurality of output image frames; and
generate an output video stream from the plurality of output image frames.
15. The system of claim 11 , wherein the instructions further cause the processor to:
generate an intervening object mask based on the intervening object pixels;
erase icon pixels within a boundary of the mask;
generate the output image frame based on the image frame and the generated icon placed in the image frame at the icon pixel coordinates;
receive a second plurality of image frames;
update the intervening object mask between each image frame from the second plurality of image frames based on object and depth information in each image frame from the second plurality of image frames;
generate a plurality of output image frames, at least in part by generating icons in each image frame based on the intervening object masks; and
generate an output video stream from the plurality of the output image frames.