Granted Patent B2

US 12,423,972 · App. 17/815,481 · Granted Sep 23, 2025

Inventors: Rebecca A. Little (Mesa, AZ); Christopher Robert Galante (Mesa, AZ)

Assignee: State Farm Mutual Automobile Insurance Company

G06V20/20G02B27/0093G02B27/0172G06F3/013G06F3/017G06Q40/08

Abstract

Claims (87)

1. A method, comprising:

receiving, by a mobile computing device comprising one or more processors, live video data representing a physical environment;

receiving, by the mobile computing device, previous image data representing the physical environment at a previous time;

identifying, by the mobile computing device and based on the live video data, a first physical object associated with a homeowners or property insurance policy;

based on identifying the first physical object associated with the homeowners or property insurance policy, retrieving, from a database, metadata associated with the first physical object;

determining, by the mobile computing device and based on the live video data, the previous image data, the homeowners or property insurance policy, and the retrieved metadata, a requested action to be performed on the first physical object;

providing, by the mobile computing device, a real-time augmented or mixed reality representation of the physical environment, including:

a virtual assistant identifying the first physical object in the physical environment; and

rendering instructions for a user to perform the requested action within the representation at a location proximate to the first physical object or the virtual assistant;

determining, based on the live video data, that the user has performed the requested action on the first physical object;

based on determining that the user has performed the requested action, capturing new image data of a surface of the first physical object exposed by the performing of the requested action; and

generating or updating, by the mobile computing device and based on the new image data, an insurance claim of the homeowners or property insurance policy pertaining to the first physical object.

2. The method of claim 1 , further comprising:

receiving a first input comprising at least one of:

receiving and processing a user utterance; or

receiving and interpreting a user body gesture using a gesture interpreter,

wherein the generating or updating the insurance claim of the homeowners or property insurance policy is further based at least in part on the first input.

3. The method of claim 1 , wherein identifying the first physical object associated with the homeowners or property insurance policy further comprises:

receiving pupil data representative of a gaze direction of a user relative to the physical environment;

identifying the first physical object based at least in part on the gaze direction; and

verifying that the first physical object is associated with the homeowners or property insurance policy.

4. The method of claim 1 , wherein the previous image data representing the physical environment at the previous time comprises point cloud data representing the physical environment.

5. The method of claim 1 , wherein providing the real-time augmented or mixed reality representation of the physical environment comprises:

determining a question relating to the first physical object, based at least in part on the live video data and a requirement of the insurance claim; and

presenting the question within the real-time augmented or mixed reality representation of the physical environment, while the virtual assistant is identifying the first physical object in the physical environment.

6. The method of claim 1 , wherein identifying the first physical object comprises:

providing the live video data to a trained image recognition machine learning engine; and

identifying the first physical object based at least in part on an output of the trained image recognition machine learning engine.

7. The method of claim 1 , wherein identifying the first physical object comprises:

determining an object identifier associated with the first physical object; and

determining the request related to the first physical object, based at least in part on the object identifier.

8. The method of claim 1 , further comprising:

automatically determining damage to the first physical object, based at least in part comparing the previous image data and the live video data; and

determining the request related to the first physical object, based at least in part on the determined damage.

9. The method of claim 1 , wherein determining the request related to the first physical object comprises:

determining a previous position of an object feature of the first physical object, based on the previous image data; and

comparing the previous position to a current position of the object feature of the first physical object, based on the current image data.

10. The method of claim 1 , wherein the request comprises a request that a user perform an action on the first physical object.

11. The method of claim 10 , further comprising:

determining that the requested action has been performed on the first physical object; and

obtaining updated current image data, based on the requested action having been performed, the updated current image data including an additional surface of the first physical object.

12. The method of claim 1 , wherein:

retrieving the metadata comprises determining a value of the first physical object; and

providing the representation comprises rendering the value of the first physical object within the representation.

13. The method of claim 1 , wherein:

retrieving the metadata comprises identifying an insurance policy covering the first physical object; and

determining the request related to the first physical object is based on the insurance policy.

14. A system, comprising:

one or more processors; and

memory coupled to the one or more processors, the memory storing instructions executable by the one or more processors to perform operations including:

receiving live video data representing a physical environment;

receiving previous image data representing the physical environment at a previous time;

identifying, based on the live video data, a first physical object associated with a homeowners or property insurance policy;

based on identifying the first physical object associated with the homeowners or property insurance policy, retrieving, from a database, metadata associated with the first physical object;

determining, based on the live video data, the previous image data, the homeowners or property insurance policy, and the retrieved metadata, a requested action to be performed on the first physical object;

providing a real-time augmented or mixed reality representation of the physical environment, including:

a virtual assistant identifying the first physical object in the physical environment; and

rendering instructions for a user to perform the requested action within the representation at a location proximate to the first physical object or the virtual assistant;

determining, based on the live video data, that the user has performed the requested action on the first physical object;

based on determining that the user has performed the requested action, capturing new image data of a surface of the first physical object exposed by the performing of the requested action; and

generating or updating, based on the new image data first input, an insurance claim of the homeowners or property insurance policy pertaining to the first physical object.

15. The system of claim 14 , the operations further comprising:

generating, based at least in part on the previous image data, an object list, the object list including an indication of the first physical object.

16. The system of claim 14 , the operations further comprising:

identifying, based at least in part on the live video data, a second physical object, wherein the real-time augmented or mixed reality representation of the physical environment includes the second physical object with the virtual assistant providing a second request related to the second physical object; and

receiving, responsive to the second request from the virtual assistant, a second input, wherein the insurance claim includes an indication of the second physical object based at least in part on the second input.

17. The system of claim 14 , wherein the virtual assistant is in a shape of a human.

18. The system of claim 14 , the operations further comprising:

receiving a first input comprising at least one of:

receiving and processing a user utterance; or

receiving and interpreting a user body gesture using a gesture interpreter,

wherein the generating or updating the insurance claim of the homeowners or property insurance policy is further based at least in part on the first input.

19. A system for generating an insurance claim, comprising:

a means for receiving live video data representing a physical environment;

a means for receiving previous image data representing the physical environment at a previous time;

a means for identifying, based on the live video data, a first physical object associated with a homeowners or property insurance policy;

a means for retrieving, from a database and based on identifying the first physical object associated with the homeowners or property insurance policy, metadata associated with the first physical object;

a means for determining, based on the live video data, the previous image data, the homeowners or property insurance policy, and the retrieved metadata, a requested action to be performed on the first physical object;

a means for providing a real-time augmented or mixed reality representation of the physical environment, including:

a virtual assistant identifying the first physical object in the physical environment; and

rendering instructions for a user to perform the requested action within the representation at a location proximate to the first physical object or the virtual assistant;

a means for determining, based on the live video data, that the user has performed the requested action on the first physical object;

a means for capturing, based on determining that the user has performed the requested action, new image data of a surface of the first physical object exposed by the performing of the requested action; and

a means for generating or updating, based on the new image data, an insurance claim of the homeowners or property insurance policy pertaining to the first physical object.

20. The system of claim 19 for generating an insurance claim, further comprising:

a means for identifying, based at least in part on the live video data, a second physical object, wherein the real-time augmented or mixed reality representation of the physical environment includes the second object with the virtual assistant providing a second request related to the second physical object; and

a means for receiving, responsive to the second request from the virtual assistant, a second input, wherein the insurance claim includes an indication of the second physical object based at least in part on the second input.

Assignments (1)

Continuity (2)

References Cited (122)