IP Library › Granted Patent US 11,080,838

Granted Patent B1

US 11,080,838 · App. 16/535,322 · Granted Aug 3, 2021

Systems and methods for image labeling using artificial intelligence

Inventors: Yuntao Li (Champaign, IL); Charlie Isaksson (Plano, TX); Marigona Bokshi-Drotar (Redmond, WA); Jing Wan (Allen, TX)

Assignee: STATE FARM MUTUAL AUTOMOBILE INSURANCE COMPANY

G06T7/0002 G06K9/00637 G06T2207/20081 G06T2207/30184

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 11,080,838
App. No.: 16/535,322
Granted: Aug 3, 2021
Kind: B1

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

An image analysis (“IA”) computer system for analyzing images of hail damage includes at least one processor in communication with at least one memory device. The at least one processor is programmed to: (i) store a damage prediction model associated with a rooftop, wherein the damage prediction model utilizes an artificial intelligence algorithm; (ii) display, to a user, an image of a rooftop; (iii) receive, from the user, a request to analyze damage to the rooftop; (iv) apply, by the at least one processor, the damage prediction model to the image, the damage prediction model outputting a plurality of damage prediction locations of the rooftop in relation to the image; and/or (v) display, by the at least one processor, an overlay box at each of the plurality of damage prediction locations, the overlay box being a virtual object overlaid onto the image for labeling the damage prediction locations.

Claims (72)

1. An image analysis (“IA”) computer system for analyzing images of hail damage using artificial intelligence, the IA computer system including at least one processor in communication with at least one memory device, the at least one processor programmed to:

store, in the at least one memory device, a damage prediction model associated with a rooftop, wherein the damage prediction model utilizes an artificial intelligence algorithm, including machine learning techniques for analyzing rooftops and identifying damage thereto;

display, to a user on a user interface of a user computing device, an image of a rooftop;

receive, from the user via the user interface of the user computing device, a request to analyze damage to the rooftop displayed in the image;

in response to the request, apply, by the at least one processor, the image to the damage prediction model, the damage prediction model outputting a plurality of damage prediction locations of the rooftop in relation to the image;

display, by the at least one processor, on the user interface of the user computing device, an overlay box at each of the plurality of damage prediction locations, the overlay box being a virtual object overlaid onto the image displayed on the user interface;

receive, from the user computing device, a user selection of a location on the image displayed on the user interface;

when the user-selected location corresponds to one overlay box at one of the plurality of damage prediction locations:

display additional information associated with the corresponding damage prediction location; and

receive, from the user computing device, a user interaction verifying whether the corresponding damage prediction location correlates to an actual damage location; and

when the user-selected location does not correspond to any overlay box:

add, in response to the user selection of the location, a new overlay box including a new virtual overlay at the user-selected location; and

determine whether the new overlay box correlates to an actual damage location.

2. The IA computer system of claim 1 , wherein the at least one processor is further programmed to, when the user-selected location corresponds to one overlay box at one of the plurality of damage prediction locations, change a color of the corresponding overlay box from a first color to a second color in response to the user interaction.

3. The IA computer system of claim 1 , wherein the at least one processor is further programmed to, when the location corresponds to one overlay box at one of the plurality of damage prediction locations, display a magnified view of the damage prediction location associated with the corresponding overlay box.

4. The IA computer system of claim 1 , wherein actual damage locations represent hail damage to the rooftop displayed in the image.

5. The IA computer system of claim 1 , wherein the at least one processor is further programmed to:

automatically determine, by the at least one processor, that the new overlay box corresponds to an actual damage location associated with hail damage; and

store, by the at least one processor, image data associated with the determination in a database of the at least one memory device.

6. The IA computer system of claim 5 , wherein the at least one processor is further programmed to automatically update the damage prediction model using the stored determination.

7. The IA computer system of claim 5 , wherein the image data includes location coordinates for the actual damage location defined by the new virtual object.

8. The IA computer system of claim 1 , wherein the at least one processor is further programmed to receive user feedback for the image, wherein the user feedback includes comments as to a quality of the image and a damage classification for each of the plurality of damage prediction locations.

9. The IA computer system of claim 1 , wherein the at least one processor is further programmed to:

assign, by the at least one processor, a first color to the overlay boxes associated with the plurality of damage prediction locations;

when the user-selected location corresponds to one overlay box at one of the plurality of damage prediction locations, update, in response to receiving the user selection, the corresponding overlay box from the first color to a second color; and

display, by the at least one processor, the updated overlay box at the corresponding damage prediction location.

10. The IA computer system of claim 1 , wherein the at least one processor is further programmed to:

collect performance data associated with the user, wherein the performance data is associated with an ability to correctly identify actual damage locations;

determine, based upon the collected performance data, a performance level of the user; and

display, to the user, a subsequent rooftop image after the user completes verification of the plurality of damage prediction locations, wherein an image quality of the subsequent rooftop image corresponds to the determined performance level.

11. The IA computer system of claim 1 , wherein the at least one processor is programmed to apply the image to the damage prediction model by:

storing, in the at least one memory device, historical image data associated with rooftop damage; and

applying a plurality of damage parameters to the image, wherein the plurality of damage parameters are determined, by the at least one processor, based upon the historical image data.

12. A computer-implemented method for analyzing images of hail damage using artificial intelligence, the method implemented using an image analysis (“IA”) computer system, the IA computer system including at least one processor in communication with at least one memory device, the method comprising:

storing, in the at least one memory device, a damage prediction model associated with a rooftop, wherein the damage prediction model utilizes an artificial intelligence algorithm, including machine learning techniques for analyzing rooftops and identifying damage thereto;

displaying, to a user on a user interface of a user computing device, an image of a rooftop;

receiving, from the user via the user interface of the user computing device, a request to analyze damage to the rooftop displayed in the image;

in response to the request, applying, by the at least one processor, the image to the damage prediction model, the damage prediction model outputting a plurality of damage prediction locations of the rooftop in relation to the image;

displaying, by the at least one processor, on the user interface of the user computing device, an overlay box at each of the plurality of damage prediction locations, the overlay box being a virtual object overlaid onto the image displayed on the user interface;

receiving, from the user computing device, a user selection of a location on the image displayed on the user interface;

when the user-selected location corresponds to one overlay box at one of the plurality of damage prediction locations:

displaying additional information associated with the corresponding damage prediction location; and

receiving, from the user computing device, a user interaction verifying whether the corresponding damage prediction location correlates to an actual damage location; and

when the user-selected location does not correspond to any overlay box:

adding, in response to the user selection of the location, a new overlay box including a new virtual overlay at the user-selected location; and

determining whether the new overlay box correlates to an actual damage location.

13. The IA computer-implemented method of claim 12 , further comprising, when the user-selected location corresponds to one overlay box at one of the plurality of damage prediction locations, changing a color of the corresponding overlay box from a first color to a second color in response to the user interaction.

14. The IA computer-implemented method of claim 12 , further comprising, when the user-selected location corresponds to one overlay box at one of the plurality of damage prediction locations, displaying a magnified view of the damage prediction location associated with the selected corresponding overlay box.

15. The IA computer-implemented method of claim 12 , wherein the actual damage location represent hail damage to the rooftop displayed in the image.

16. The IA computer-implemented method of claim 12 , further comprising:

automatically determining, by the at least one processor, that the new overlay box corresponds to an actual damage location associated with hail damage; and

storing, by the at least one processor, image data associated with the determination in a database of the at least one memory device.

17. The IA computer-implemented method of claim 16 , further comprising automatically updating the damage prediction model using the stored determination.

18. At least one non-transitory computer-readable storage media having computer-executable instructions embodied thereon, wherein when executed by at least one processor, the computer-executable instructions cause the at least one processor to:

store a damage prediction model associated with a rooftop, wherein the damage prediction model utilizes an artificial intelligence algorithm, including machine learning techniques for analyzing rooftops and identifying damage thereto;