IP Library Granted Patent US 11,448,518
Granted Patent B2
US 11,448,518 · App. 16/745,717 · Granted Sep 20, 2022

Augmented reality navigational overlay

Inventors: Chen-Ping Yu (Santa Clara, CA); Xinyu Li (Jersey City, NY); Aaditya Chandrasekhar (San Francisco, CA)
G01C21/3647G01C21/3415G01C21/365G01C21/3691G06T7/11G06T11/60G06T2207/30252
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,448,518
App. No.
16/745,717
Granted
Sep 20, 2022
Kind
B2
Abstract

A system and method may generate a more realistic augmented reality (AR) overlay by generating a segmentation image and blending it with one or more other images. The system may generate a segmentation image based on an input image. The segmentation image may be blended with an AR path overlay image to generate an object-masked AR path overlay image. The object-masked AR path overlay image may be blended with the input image to generate an output image.

Claims (39)

1. A method, comprising:

generating a guidance indicator corresponding to a guidance route;

obtaining an image of the guidance route including an object;

generating a segmentation image based on the image, the segmentation image having first pixels corresponding to the object in the image;

modifying a portion of the guidance indicator based on at least a position of the first pixels in the segmentation image by occluding the portion of the guidance indicator from first pixels; and

displaying the modified guidance indicator on a display.

2. The method of claim 1 , wherein generating the segmentation image is performed by applying semantic segmentation on the image.

3. The method of claim 2 , wherein the image comprises an image of a roadway, and wherein the at least one object comprises at least one object on the roadway.

4. The method of claim 3 , wherein generating the guidance indicator associated with the navigational guidance route comprises generating an augmented-reality path overlay image including the guidance indicator.

5. The method of claim 4 , wherein modifying the guidance indicator based on the at least one object in the image comprises modifying pixels in the augmented-reality path overlay image that spatially correspond to the first pixels in the segmentation image.

6. The method of claim 5 , wherein displaying the modified guidance indicator in association with the navigational guidance route comprises generating an output image by combining the image with the augmented-reality path overlay image having the modified guidance indicator.

7. The method of claim 6 , further comprising displaying the output image on a display of a mobile electronic device.

8. The method of claim 5 , wherein displaying the modified guidance indicator in association with the navigational guidance route comprises projecting the augmented-reality path overlay image having the modified guidance indicator onto a windshield of a vehicle.

9. The method of claim 1 , further comprising determining the navigational guidance route based on a global positioning system location and map information.

10. A navigation system comprising one or more non-transitory computer-readable media storing computer-executable instructions that, when executed on one or more processors, cause the one or more processors to perform acts comprising:

generating a guidance indicator corresponding to a guidance route;

obtaining an image of the guidance route including an object;

generating a segmentation image based on the image, the segmentation image having first pixels corresponding to the object in the image;

modifying a portion of the guidance indicator based on at least a position of the first pixels in the segmentation image; and

displaying the modified guidance indicator on a display.

11. The navigation system of claim 10 , wherein generating the segmentation image is performed by applying semantic segmentation on the image.

12. The navigation system of claim 11 , wherein the image comprises an image of a roadway, and wherein the at least one object comprises at least one object on the roadway.

13. The navigation system of claim 12 , wherein generating the guidance indicator associated with the navigational guidance route comprises generating an augmented-reality path overlay image including the guidance indicator.

14. The navigation system of claim 13 , wherein modifying the guidance indicator based on the at least one object in the image comprises modifying pixels in the augmented-reality path overlay image that spatially correspond to the first pixels in the segmentation image.

15. The navigation system of claim 14 , wherein displaying the modified guidance indicator in association with the navigational guidance route comprises generating an output image by combining the image with the augmented-reality path overlay image having the modified guidance indicator.

16. The navigation system of claim 15 , further comprising instructions for displaying the output image on a display of a mobile electronic device.

17. The navigation system of claim 16 , wherein displaying the modified guidance indicator in association with the navigational guidance route comprises projecting the augmented-reality path overlay image having the modified guidance indicator onto a windshield of a vehicle.

18. A method for generating a path overlay, the method comprising:

obtaining an image of the guidance route;

generating a guidance indicator corresponding to a guidance route, the guidance indicator comprising a plurality of pixels;

detecting a first object in the path of the guidance indicator;

generating a segmentation image based on the image, the segmentation image having first pixels corresponding to the first object in the image;

modifying the guidance indicator by occluding a portion of the guidance indicator by removing or dimming a portion of the plurality of pixels of the guidance indicator; and

displaying the modified guidance indicator on a display.

19. The method of claim 18 , wherein the first object detected is a non-moving vehicle.

20. The method of claim 18 , wherein the first object detected is a stationary object in the environment.

21. The method of claim 20 , wherein the stationary object is one of a tree, sign, pedestrian, physical obstruction in the environment, traffic light, curb, building, bridge.

22. The method of claim 18 , wherein the display is operably coupled to a moving vehicle, and wherein the first object detected is a portion of the moving vehicle.

23. The method of claim 22 , wherein the portion of the moving vehicle is a hood portion of the vehicle such that the occluding of the portion of the guidance indicator is occluded by the hood portion of the vehicle.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 30, 2022
From: PHIAR TECHNOLOGIES, INC.
To: GOOGLE LLC
Reel/Frame 061919/0013 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 29, 2020
From: YU, CHEN-PING; LI, XINYU; CHANDRASEKHAR, AADITYA
To: PHIAR TECHNOLOGIES, INC.
Reel/Frame 051648/0701 →
Continuity (4)
Continuation In Part 16569650 · Sep 12, 2019
Continuation 16255825 · Jan 24, 2019
Provisional Application 62737563 · Sep 27, 2018
Related Publication 20200149916A1 · May 14, 2020