IP Library › Granted Patent US 11,468,642
Granted Patent B2
US 11,468,642 · App. 16/922,936 · Granted Oct 11, 2022

Use of augmented reality for service projects

Inventors: Yuhua Xie (Walnut Creek, CA); Amanda Burgmeier (Saratoga, CA); Jin Salil Barai (Ontario, CA); Athitya Kumar (Bangalore, IN)
Assignee: Intuit Inc.
G06T19/006
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,468,642
App. No.
16/922,936
Filed
Jul 7, 2020
Granted
Oct 11, 2022
Kind
B2
Art Unit
2613
USPC
345/633
Abstract

Aspects of using augment reality for service projects are described. In some implementations, an augmented reality (AR) system receives one or more images of a real-world space, detects a physical object in the real-world space from the one or more images, and identifies the physical object for a service project. The AR system also generates a virtual object associated with the identified physical object and outputs the virtual object for display in an AR environment. The AR system further receives a selection of one or more portions of the virtual object associated with one or more portions of the physical object to be serviced and generates information for performing the service project based on the one or more portions of the physical object.

Claims (91)

1. A method performed by one or more processors of an augmented reality (AR) system, the method comprising:

displaying, on a touch sensitive display of the AR system, a control menu, wherein the control menu includes one or more buttons to allow a user to switch the AR system between:

an identification and diagnostic mode to identify which portions of a physical object are to be serviced; and

an estimate mode to generate an estimate;

while in the identification and diagnostic mode:

receiving one or more images of a real-world space;

detecting the physical object in the real-world space from the one or more images;

identifying the physical object for a service project;

generating a virtual object associated with the physical object, wherein the virtual object includes one or more parts that are associated with one or more parts of the physical object that are not visible in the real-world space;

displaying the virtual object in an AR environment on the touch sensitive display, wherein the virtual object includes a plurality of parts selectable by the user, the plurality of parts including the one or more parts of the virtual object associated with the one or more parts of the physical object that are not visible in the real-world space;

receiving, at the touch sensitive display, a user selection of one or more parts of the virtual object associated with one or more parts of the physical object to be serviced; and

highlighting, on the touch sensitive display, the one or more parts of the virtual object associated with the one or more parts of the physical object to be serviced as selected by the user;

receiving, at the touch sensitive display, an input to the control menu to switch the AR system from the identification and diagnostic mode to the estimate mode; and

while in the estimate mode:

generating a parts list for the service project based on the one or more parts of the physical object to be serviced;

generating a total estimate for the service project based on the parts list, wherein generating the total estimate includes obtaining a cost for each part of the one or more parts of the physical object to be serviced and obtaining a labor estimate; and

displaying, on the touch sensitive display, the total estimate for the service project.

2. The method of claim 1 , wherein generating the total estimate for the service project is based on:

one or more prior service project records for the physical object.

3. The method of claim 1 , further comprising outputting the total estimate for the service project for transmission to a computer system for managing service project records.

4. The method of claim 3 , further comprising:

receiving, at the touch sensitive display, another input to the control menu to switch the AR system from one of the identification and diagnostic mode or the estimate mode to an invoice mode to generate an invoice for the service project, wherein the invoice mode is associated with a button of the one or more buttons of the control menu; and

while in the invoice mode:

generating the invoice for the service project based on the total estimate for the service project;

displaying, on the touch sensitive display, the invoice for the service project; and

outputting the invoice for the service project for transmission to the computer system for managing the service project records.

5. The method of claim 1 , wherein identifying the physical object includes automatically identifying one or more of a type, a make, or a model of the physical object without user input.

6. The method of claim 1 , further comprising displaying the virtual object concurrently with at least one image of the one or more images in the AR environment.

7. The method of claim 1 , further comprising displaying the virtual object superimposed on the physical object from a view of the user of the real-world space.

8. The method of claim 1 , further comprising obtaining one or more assembly diagrams associated with the physical object, wherein:

the virtual object is based on the one or more assembly diagrams; and

the one or more assembly diagrams include the one or more parts of the virtual object associated with the one or more parts of the physical object that are not visible in the real-world space.

9. An augmented reality (AR) system comprising:

one or more cameras;

one or more displays, wherein the one or more displays includes a touch sensitive display;

one or more processors; and

a memory storing instructions that, when executed by the one or more processors, cause the AR system to:

display, on the touch sensitive display, a control menu, wherein the control menu includes one or more buttons to allow a user to switch the AR system between:

an identification and diagnostic mode to identify which portions of a physical object are to be serviced; and

an estimate mode to generate an estimate;

while in the identification and diagnostic mode:

capture one or more images of a real-world space by the one or more cameras;

detect the physical object in the real-world space from the one or more images;

identify the physical object for a service project;

generate a virtual object associated with the physical object, wherein the virtual object includes one or more parts that are associated with one or more parts of the physical object that are not visible in the real-world space;

display the virtual object in an AR environment on the touch sensitive display, wherein the virtual object includes a plurality of parts selectable by the user, the plurality of parts including the one or more parts of the virtual object associated with the one or more parts of the physical object that are not visible in the real-world space;

receive, at the touch sensitive display, a user selection of one or more parts of the virtual object associated with one or more parts of the physical object to be serviced; and

highlight, on the touch sensitive display, the one or more parts of the virtual object associated with the one or more parts of the physical object to be serviced as selected by the user;

receive, at the touch sensitive display, an input to the control menu to switch the AR system from the identification and diagnostic mode to the estimate mode; and

while in the estimate mode:

generate a parts list for the service project based on the one or more parts of the physical object to be serviced;

generate a total estimate for the service project based on the parts list, wherein generating the total estimate includes obtaining a cost for each part of the one or more parts of the physical object to be serviced and obtaining a labor estimate; and

display, on the touch sensitive display, the total estimate for the service project.

10. The AR system of claim 9 , wherein generating the total estimate for the service project is based on:

one or more prior service project records for the physical object.

11. The AR system of claim 9 , further comprising a transceiver, wherein execution of the instructions further causes the AR system to transmit, by the transceiver, the total estimate for the service project to a computer system for managing service project records.

12. The AR system of claim 11 , wherein execution of the instructions further causes the AR system to:

receive, at the touch sensitive display, another input to the control menu to switch the AR system from one of the identification and diagnostic mode or the estimate mode to an invoice mode to generate an invoice for the service project, wherein the invoice mode is associated with a button of the one or more buttons of the control menu; and

while in the invoice mode:

generate the invoice for the service project based on the total estimate for the service project;

display, on the touch sensitive display, the invoice for the service project; and

transmit, by the transceiver, the invoice for the service project to the computer system for managing the service project records.

13. The AR system of claim 9 , wherein identifying the physical object includes automatically identifying one or more of a type, a make, or a model of the physical object without user input.

14. The AR system of claim 9 , wherein displaying the virtual object in the AR environment includes displaying the virtual object concurrently with at least one image of the one or more images.

15. The AR system of claim 9 , wherein displaying the virtual object in the AR environment includes displaying the virtual object superimposed on the physical object from a view of the user of the real-world space.

16. The AR system of claim 9 , wherein execution of the instructions further causes the AR system to obtain one or more assembly diagrams associated with the physical object, wherein:

the virtual object is based on the one or more assembly diagrams; and

the one or more assembly diagrams include the one or more parts of the virtual object associated with the one or more parts of the physical object that are not visible in the real-world space.

17. A non-transitory computer readable medium storing instructions that, when executed by one or more processors of an augmented reality (AR) system, cause the AR system to:

display, on a touch sensitive display of the AR system, a control menu, wherein the control menu includes one or more buttons to allow a user to switch the AR system between:

an identification and diagnostic mode to identify which portions of a physical object are to be serviced; and

an estimate mode to generate an estimate;

while in the identification and diagnostic mode:

capture one or more images of a real-world space;

detect the physical object in the real-world space from the one or more images;

identify the physical object for a service project;

generate a virtual object associated with the physical object, wherein the virtual object includes one or more parts that are associated with one or more parts of the physical object that are not visible in the real-world space;

display the virtual object in an AR environment on the touch sensitive display, wherein the virtual object includes a plurality of parts selectable by the user, the plurality of parts including the one or more parts of the virtual object associated with the one or more parts of the physical object that are not visible in the real-world space;

receive, at the touch sensitive display, a user selection of one or more parts of the virtual object associated with one or more parts of the physical object to be serviced; and

highlight, on the touch sensitive display, the one or more parts of the virtual object associated with the one or more parts of the physical object to be serviced as selected by the user;

receive, at the touch sensitive display, an input to the control menu to switch the AR system from the identification and diagnostic mode to the estimate mode; and

while in the estimate mode:

generate a parts list for the service project based on the one or more parts of the physical object to be serviced;

generate a total estimate for the service project based on the parts list, wherein generating the total estimate includes obtaining a cost for each part of the one or more parts of the physical object to be serviced and obtaining a labor estimate; and

display, on the touch sensitive display, the total estimate for the service project.

18. The non-transitory computer readable medium of claim 17 , wherein generating the total estimate for the service project is based on:

one or more prior service project records for the physical object.

19. The non-transitory computer readable medium of claim 17 , wherein identifying the physical object includes automatically identifying one or more of a type, a make, or a model of the physical object without user input.

20. The non-transitory computer readable medium of claim 17 , wherein execution of the instructions further causes the AR system to obtain one or more assembly diagrams associated with the physical object, wherein:

the virtual object is based on the one or more assembly diagrams; and

the one or more assembly diagrams include the one or more parts of the virtual object associated with the one or more parts of the physical object that are not visible in the real-world space.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 8, 2020
From: XIE, YUHUA; BURGMEIER, AMANDA; BARAI, JIN SALIL; KUMAR, ATHITYA
To: INTUIT INC.
Reel/Frame 053148/0874 →
Continuity (1)
Related Publication 20220012947A1 · Jan 13, 2022
Cited By (1)
US 12,638,963