IP Library Granted Patent US 11,132,850
Granted Patent B1
US 11,132,850 · App. 16/827,954 · Granted Sep 28, 2021

Augmented reality diagnostic interface

Inventors: Fabio Minoru Tanada (Campinas, BR); Edson Gomes Pereira (Cotia, BR); Patricia Pereira Fantini de Melo (Campinas, BR); Fatima Teixeira Pombo Barile (Campinas, BR); Maria Do Carmo Miranda Barbosa (Campinas, BR)
Assignee: International Business Machines Corporation
G07C5/008G06T11/60G07C5/006
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,132,850
App. No.
16/827,954
Granted
Sep 28, 2021
Kind
B1
Abstract

A method, system, and computer program product for augmented reality-based vehicle diagnostics are provided. The method receives vehicle metadata describing a vehicle. The vehicle metadata includes a problem context. The method retrieves vehicle reference data based on the vehicle metadata. A candidate problem is generated for the problem context based on the vehicle metadata and the vehicle reference data. The method generates a set of candidate solutions for the candidate problem. The set of candidate solutions address the problem context. The method generates a set of visualizations for the set of solutions based on the vehicle metadata and the candidate problem. The method presents a visualization of the set of visualizations as an augmented reality user interface.

Claims (79)

1. A computer-implemented method, comprising:

receiving vehicle metadata describing a vehicle from a mobile computing device, the vehicle metadata including a problem context;

based on the vehicle metadata, retrieving vehicle reference data;

based on the vehicle metadata and the vehicle reference data, generating a candidate problem for the problem context;

generating a set of candidate solutions for the candidate problem, the set of candidate solutions configured to address the problem context;

based on the vehicle metadata and the candidate problem, generating a set of visualizations for the set of candidate solutions, each visualization generated for a specified solution based on the vehicle reference data; and

presenting, on a display device of the mobile computing device, a visualization of the set of visualizations as an augmented reality user interface.

2. The computer-implemented method of claim 1 , further comprising:

retrieving functionality data describing a functional state of the vehicle;

based on the functionality data, retrieving experiential data describing user experiences with one or more similar vehicles, the one or more similar vehicles sharing one or more vehicle characteristic with the vehicle; and

based on the functionality data and the experiential data, generating a set of vehicle reference data for the vehicle.

3. The computer-implemented method of claim 2 , wherein the functionality data and the experiential data are non-structured data, and generating the set of vehicle reference data for the vehicle further comprises:

normalizing the functionality data and the experiential data, based on a data structure for the vehicle reference data, to generate normalized vehicle data; and

generating the set of vehicle reference data from the normalized vehicle data.

4. The computer-implemented method of claim 1 , wherein the candidate problem is a set of candidate problems and the set of candidate solutions is a plurality of sets of candidate solutions, each set of candidate solutions being generated for a specified candidate problem of the set of candidate problems.

5. The computer-implemented method of claim 1 , wherein generating the set of candidate solutions further comprises:

based on the vehicle metadata and the vehicle reference data, generating an augmented instruction set for a candidate solution of the set of candidate solutions;

receiving one or more images responsive to the augmented instruction set, the one or more images depicting a portion of the vehicle associated with the candidate problem; and

selecting a candidate solution, from the set of candidate solutions, based on the one or more images.

6. The computer-implemented method of claim 1 , wherein generating the set of visualizations further comprises:

for each visualization, identifying one or more reference images for a candidate solution associated with the visualization, the one or more reference images selected from the vehicle reference data; and

modifying the one or more reference images for presentation within the augmented reality user interface.

7. The computer-implemented method of claim 1 , wherein presenting the visualization as an augmented reality user interface further comprises:

initializing an image capture device of the mobile computing device, the image capture device depicting a field of view on the display device;

detecting a vehicle component within the field of view of the image capture device; and

generating one or more overlay elements on the display device, the one or more overlay elements providing solution information for a candidate solution associated with the visualization.

8. A system, comprising:

one or more processors; and

a computer-readable storage medium, coupled to the one or more processors, storing program instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

receiving vehicle metadata describing a vehicle from a mobile computing device, the vehicle metadata including a problem context;

based on the vehicle metadata, retrieving vehicle reference data;

based on the vehicle metadata and the vehicle reference data, generating a candidate problem for the problem context;

generating a set of candidate solutions for the candidate problem, the set of candidate solutions configured to address the problem context;

based on the vehicle metadata and the candidate problem, generating a set of visualizations for the set of candidate solutions, each visualization generated for a specified solution based on the vehicle reference data; and

presenting, on a display device of the mobile computing device, a visualization of the set of visualizations as an augmented reality user interface.

9. The system of claim 8 , wherein the operations further comprise:

retrieving functionality data describing a functional state of the vehicle;

based on the functionality data, retrieving experiential data describing user experiences with one or more similar vehicles, the one or more similar vehicles sharing one or more vehicle characteristics with the vehicle; and

based on the functionality data and the experiential data, generating a set of vehicle reference data for the vehicle.

10. The system of claim 9 , wherein the functionality data and the experiential data are non-structured data, and generating the set of vehicle reference data for the vehicle further comprises:

normalizing the functionality data and the experiential data, based on a data structure for the vehicle reference data, to generate normalized vehicle data; and

generating the set of vehicle reference data from the normalized vehicle data.

11. The system of claim 8 , wherein the candidate problem is a set of candidate problems and the set of candidate solutions is a plurality of sets of candidate solutions, each set of candidate solutions being generated for a specified candidate problem of the set of candidate problems.

12. The system of claim 8 , wherein generating the set of candidate solutions further comprises:

based on the vehicle metadata and the vehicle reference data, generating an augmented instruction set for a candidate solution of the set of candidate solutions;

receiving one or more images responsive to the augmented instruction set, the one or more images depicting a portion of the vehicle associated with the candidate problem; and

selecting a candidate solution, from the set of candidate solutions, based on the one or more images.

13. The system of claim 8 , wherein generating the set of visualizations further comprises:

for each visualization, identifying one or more reference images for a candidate solution associated with the visualization, the one or more reference images selected from the vehicle reference data; and

modifying the one or more reference images for presentation within the augmented reality user interface.

14. The system of claim 8 , wherein presenting the visualization as an augmented reality user interface further comprises:

initializing an image capture device of the mobile computing device, the image capture device depicting a field of view on the display device;

detecting a vehicle component within the field of view of the image capture device; and

generating one or more overlay elements on the display device, the one or more overlay elements providing solution information for a candidate solution associated with the visualization.

15. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions being executable by one or more processors to cause the one or more processors to perform operations comprising:

receiving vehicle metadata describing a vehicle from a mobile computing device, the vehicle metadata including a problem context;

based on the vehicle metadata, retrieving vehicle reference data;

based on the vehicle metadata and the vehicle reference data, generating a candidate problem for the problem context;

generating a set of candidate solutions for the candidate problem, the set of candidate solutions configured to address the problem context;

based on the vehicle metadata and the candidate problem, generating a set of visualizations for the set of candidate solutions, each visualization generated for a specified solution based on the vehicle reference data; and

presenting, on a display device of the mobile computing device, a visualization of the set of visualizations as an augmented reality user interface.

16. The computer program product of claim 15 , wherein the operations further comprise:

retrieving functionality data describing a functional state of the vehicle;

based on the functionality data, retrieving experiential data describing user experiences with one or more similar vehicles, the one or more similar vehicles sharing one or more vehicle characteristics with the vehicle; and

based on the functionality data and the experiential data, generating a set of vehicle reference data for the vehicle.

17. The computer program product of claim 16 , wherein the functionality data and the experiential data are non-structured data, and generating the set of vehicle reference data for the vehicle further comprises:

normalizing the functionality data and the experiential data, based on a data structure for the vehicle reference data, to generate normalized vehicle data; and

generating the set of vehicle reference data from the normalized vehicle data.

18. The computer program product of claim 15 , wherein generating the set of candidate solutions further comprises:

based on the vehicle metadata and the vehicle reference data, generating an augmented instruction set for a candidate solution of the set of candidate solutions;

receiving one or more images responsive to the augmented instruction set, the one or more images depicting a portion of the vehicle associated with the candidate problem; and

selecting a candidate solution, from the set of candidate solutions, based on the one or more images.

19. The computer program product of claim 15 , wherein generating the set of visualizations further comprises:

for each visualization, identifying one or more reference images for a candidate solution associated with the visualization, the one or more reference images selected from the vehicle reference data; and

modifying the one or more reference images for presentation within the augmented reality user interface.

20. The computer program product of claim 15 , wherein presenting the visualization as an augmented reality user interface further comprises:

initializing an image capture device of the mobile computing device, the image capture device depicting a field of view on the display device;

detecting a vehicle component within the field of view of the image capture device; and

generating one or more overlay elements on the display device, the one or more overlay elements providing solution information for a candidate solution associated with the visualization.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 18, 2021
From: INTERNATIONAL BUSINESS MACHINES CORPORATION
To: KYNDRYL, INC.
Reel/Frame 058213/0912 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 24, 2020
From: TANADA, FABIO MINORU; GOMES PEREIRA, EDSON; PEREIRA FANTINI DE MELO, PATRICIA; TEIXEIRA POMBO BARILE, FATIMA; MIRANDA BARBOSA, MARIA DO CARMO
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 052207/0831 →