Artificial intelligence based determination of damage to physical structures via video

1. A method for assessing damage to a structure, the method comprising:

training a first machine learning (ML) module to identify external parts of various structures from frames of a first plurality of videos of the various structures;

training a second machine learning (ML) module to detect and classify damaged regions of the various structures from frames of a second plurality of videos of the various structures;

using the first ML module to detect one or more external parts of the structure from one or more frames of a video of the structure;

using the second ML module to: (i) determine one or more damaged regions of the structure and associated severity levels, (ii) remove a subgroup of frames of the video depicting a damaged region with a specific severity level, and (iii) classify remaining damaged regions in remaining frames of the video of the structure based on damage types;

associating the remaining damaged regions and corresponding damage types with the one or more external parts, to provide a respective vision-based damage estimate for each of the one or more external parts;

receiving audio content associated with the video of the structure;

generating text from the audio content using a speech-to-text engine;

dividing the video into a plurality of subsets of video frames, each of the subsets of video frames associated with a different window of time within the video; and

for each different window of time associated with the subsets of video frames:

identifying a portion of the text corresponding to the applicable window of time,

synchronizing the portion of text with the subset of video frames associated with the applicable window of time, and

determining whether the synchronized portion of text corroborates results of using the second ML module to classify damaged regions of the structure depicted in the subset of video frames associated with the applicable window of time.

2. The method of claim 1 , wherein the first ML module or the second ML module is trained to perform pixel level segmentation, wherein the segmentation marks the external part or the damaged region using a bounding box.

3. The method of claim 1 , further comprising:

receiving the video of the structure, the video comprising a first plurality of frames; and

selecting a second plurality of frames via down-sampling the video.

4. The method of claim 3 , further comprising:

using a classifier to partition the second plurality of frames into a set of relevant frames and a set of non-relevant frames, the relevant frames being associated with at least a portion of the structure and the non-relevant frames being devoid of a significant portion of the structure.

5. The method of claim 4 , wherein:

the set of relevant frames comprises a first relevant frame and a second relevant frame;

a first viewing angle is associated with the first relevant frame and a second viewing angle is associated with the second relevant frame; or

a first zoom factor is associated with the first relevant frame and a second zoom factor is associated with the second relevant frame.

6. The method of claim 4 , wherein the set of relevant frames comprises a first subset of relevant frames and a second subset of relevant frames, the method further comprising:

deriving using the first subset of relevant frames a first damage estimate identifying a part and a damage type;

deriving using the second subset of relevant frames a second damage estimate identifying the part and the damage type; and

aggregating the first and second damage estimates to provide a final vision-based damage estimate.

7. The method of claim 6 , wherein:

aggregating the first and second damage estimates comprises computing an average of the first and second damage estimates or a consensus of the first and second damage estimates.

8. The method of claim 6 , wherein the set of relevant frames comprises a third subset of relevant frames, the method further comprising:

deriving using the third subset of relevant frames a third damage estimate identifying the part and the damage type; and

aggregating the first, second, and third damage estimates to provide the final vision-based damage estimate,

wherein aggregating the first, second, and third damage estimates comprises computing an average, or a consensus, or a majority of the first, second, and third damage estimates.

9. The method of claim 6 , wherein:

the first damage estimate is associated with a first confidence score;

the second damage estimate is associated with a second confidence score; and

aggregating the first and second damage estimates comprises computing a weighted average of the first and second damage estimates based on the first and second confidence scores.

10. The method of claim 1 , further comprising:

identifying relevant keywords or phrases from the text;

associating the keywords or phrases with part names and damage types to obtain respective audio-based damage estimates for the one or more external parts; and

consolidating the vision-based damage estimate and the audio-based damage estimate, to provide a consolidated damage estimate.

11. The method of claim 1 , wherein the second ML module is trained to perform segmentation based on a plurality of classes of damage types, the damage types comprising: scratch, dent, tear, hole, misaligned part, missing part, crumple, or crush damage.

12. The method of claim 1 , further comprising:

generating a cost estimate for repair or replacement of at least one of the one or more external parts.

13. A system for assessing damage to a structure, the system comprising:

one or more processors; and

one or more memories in communication with one or more of the processors and comprising instructions which, when executed by one or more of the processors, program the one or more of the processors to:

train a first machine learning (ML) module to identify external parts of various structures from frames of a first plurality of videos of the various structures;

train a second machine learning (ML) module to detect and classify damaged regions of the various structures from frames of a second plurality of videos of the various structures;

operate as or use the first ML module to detect one or more external parts of the structure from one or more frames of a video of the structure;

operate as or use the second ML module to: (i) determine one or more damaged regions of the structure and associated severity levels, (ii) remove a subgroup of frames of the video depicting a damaged region with a specific severity level, and (iii) classify remaining damaged regions in remaining frames of the video of the structure based on damage types;

associate the remaining damaged regions and corresponding damage types with the one or more external parts, to provide a respective vision-based damage estimate for each of the one or more external parts;

receive audio content associated with the video of the structure;

generate text from the audio content using a speech-to-text engine;

divide the video into a plurality of subsets of video frames, each of the subsets of video frames associated with a different window of time within the video; and

for each different window of time associated with the subsets of video frames:

identify a portion of the text corresponding to the applicable window of time,

synchronize the portion of text with the subset of video frames associated with the applicable window of time, and

determine whether the synchronized portion of text corroborates results of using the second ML module to classify damaged regions of the structure depicted in the subset of video frames associated with the applicable window of time.

14. The system of claim 13 , wherein the first or second ML module is trained to perform pixel level segmentation, wherein the segmentation marks the external part or the damaged region using a bounding box.

15. The system of claim 13 , wherein the instructions further program the processor to:

receive the video of the structure, the video comprising a first plurality of frames; and

select a second plurality of frames via down-sampling the video.

16. The system of claim 15 , wherein the instructions further program the processor to:

operate as or use a classifier to partition the second plurality of frames into a set of relevant frames and a set of non-relevant frames, the relevant frames being associated with at least a portion of the structure and the non-relevant frames being devoid of a significant portion of the structure.

17. The system of claim 16 , wherein:

the set of relevant frames comprises a first relevant frame and a second relevant frame;

a first viewing angle is associated with the first relevant frame and a second viewing angle is associated with the second relevant frame; or

a first zoom factor is associated with the first relevant frame and a second zoom factor is associated with the second relevant frame.

18. The system of claim 16 , wherein:

the set of relevant frames comprises a first subset of relevant frames and a second subset of relevant frames; and

the instructions further program the processor to:

derive using the first subset of relevant frames a first damage estimate identifying a part and a damage type;

derive using the second subset of relevant frames a second damage estimate identifying the part and the damage type; and

aggregate the first and second damage estimates to provide a final vision-based damage estimate.

19. The system of claim 18 , wherein to aggregate the first and second damage estimates, the instructions program the processor to:

compute: (i) an average of the first and second damage estimates, or (ii) a consensus of the first and second damage estimates.

20. The system of claim 18 , wherein:

the set of relevant frames comprises a third subset of relevant frames; and

the instructions further program the processor to:

derive using the third subset of relevant frames a third damage estimate identifying the part and the damage type; and

aggregate the first, second, and third damage estimates to provide the final vision-based damage estimate, wherein to aggregate the first, second, and third damage estimates the instructions program the processor to compute: (i) an average, or (ii) a consensus, or (iii) a majority of the first, second, and third damage estimates.

21. The system of claim 18 , wherein:

the first damage estimate is associated with a first confidence score;

the second damage estimate is associated with a second confidence score; and

to aggregate the first and second damage estimates, the instructions program the processor to compute a weighted average of the first and second damage estimates based on the first and second confidence scores.

22. The system of claim 13 , wherein:

the instructions further program the processor to:

identify relevant keywords or phrases from the text;

associate the keywords or phrases with part names and damage types to obtain respective audio-based damage estimates for the one or more external parts; and

consolidate the vision-based damage estimate and the audio-based damage estimate, to provide a consolidated damage estimate.

23. The system of claim 13 , wherein the second ML module is trained to perform segmentation based on a plurality of classes of damage types, the damage types comprising:

scratch, dent, tear, hole, misaligned part, missing part, crumple, or crush damage.

24. The system of claim 13 , wherein the instructions further program the processor to:

generate a cost estimate for repair or replacement of at least one of the one or more external parts.