Automated foreign object debris detection system using generative AI

1 . A computer-implemented method for detecting foreign object debris (FOD), the method comprising:

using a number of processors to perform:

receiving images captured by a number of imaging devices in a defined inspection area;

generating, by a pre-trained generative artificial intelligence (AI) model, a natural language text caption in complete sentences describing the images to enable easier interpretation of objects in the images, wherein the caption is newly generated and includes a position and condition of the objects in relation to their immediate surroundings in the images;

classifying, by a text classifier AI model, FOD in the images based on the natural language text caption, and a confidence determination of whether FOD is present in the natural language text caption;

identifying, by an explainable AI model, words and phrases within the natural language text caption according to which the text classifier AI model made the classification and the confidence determination of whether FOD is present in the natural language text caption; and

displaying, in a user interface, a report including, for each identified FOD, a location where the FOD was identified in the inspection area, the captured image with the natural language text caption, and color coded highlighting of the key words and phrases identified by the explainable AI model to determine whether FOD is present in the natural language text caption.

2 . The method of claim 1 , wherein the images are captured by:

receiving, from a tracking device, positional tracking data of a user moving through the defined inspection area; and

directing the number of imaging devices along a path through the inspection area defined by the positional tracking data to capture the images.

3 . The method of claim 2 , wherein the tracking device comprises one of:

a wrist tracker;

a smart watch; or

a handheld mobile device.

4 . The method of claim 1 , further comprising translating, by a generative AI language translation model, the natural language explanations into a specified second language.

5 . The method of claim 1 , wherein the imaging devices comprises at least one of:

Simultaneous Localization and Mapping devices;

drones; or

static cameras.

6 . The method of claim 1 , wherein the captured images comprise at least one of:

still images; or

video.

7 . The method of claim 1 , wherein the explainable AI model comprises at least one of:

a SHapley Additive explanations model;

a Local Interpretable Model Agnostic Explanation model;

tree surrogates;

Global Interpretation via Recursive Partitioning;

Explainable Boosting Machine; or

Contrastive Explanation Method.

8 . The method of claim 1 , wherein the confidence determination of whether FOD is present in the natural language text caption comprises a determination of whether a confidence score of whether or not there is FOD in the images based on the natural language text caption is above a set threshold.

9 . A system for detecting foreign object debris (FOD), the system comprising:

a storage device that stores program instructions;

one or more processors operably connected to the storage device and configured to execute the program instructions to cause the system to:

receive images captured by a number of imaging devices in a defined inspection area;

generate, by a pre-trained generative artificial intelligence (AI) model, a natural language text caption describing the images to enable easier interpretation of objects in the images, wherein the caption is newly generated and includes a position and condition of the objects in relation to their immediate surroundings in the images;

classify, by a text classifier AI model, FOD in the images based on the natural language text caption, and a confidence determination of whether FOD is present in the natural language text caption;

identify, by an explainable AI model, words and phrases within the natural language text caption according to which the text classifier AI model made the classification and the confidence determination of whether FOD is present in the natural language text caption; and

display, in a user interface, a report including, for each identified FOD, a location where the FOD was identified in the inspection area, the captured image with the natural language text caption, and color coded highlighting of the key words and phrases identified by the explainable AI model to determine whether FOD is present in the natural language text caption.

10 . The system of claim 9 , wherein the images are captured by:

receiving, from a tracking device, positional tracking data of a user moving through the defined inspection area; and

directing the number of imaging devices along a path through the inspection area defined by the positional tracking data to capture the images.

11 . The system of claim 10 , wherein the tracking device comprises one of:

a wrist tracker;

a smart watch; or

a handheld mobile device.

12 . The system of claim 9 , wherein the processors further execute instructions to translate, by a generative AI language translation model, the natural language explanations into a specified second language.

13 . The system of claim 9 , wherein the imaging devices comprises at least one of:

Simultaneous Localization and Mapping devices;

drones; or

static cameras.

14 . The system of claim 9 , wherein the captured images comprise at least one of:

still images; or

video.

15 . The system of claim 9 , wherein the explainable AI model comprises at least one of:

a SHapley Additive explanations model;

a Local Interpretable Model Agnostic Explanation model;

tree surrogates;

Global Interpretation via Recursive Partitioning;

Explainable Boosting Machine; or

Contrastive Explanation Method.

16 . A computer program product for detecting foreign object debris (FOD), the computer program product comprising:

a non-transitory computer-readable storage medium having program instructions embodied thereon to perform the operations of:

receiving images captured by a number of imaging devices in a defined inspection area;

generating, by a pre-trained generative artificial intelligence (AI) model, a natural language text caption describing the images to enable easier interpretation of objects in the images, wherein the caption is newly generated and includes a position and condition of the objects in relation to their immediate surroundings in the images;

classifying, by a text classifier AI model, FOD in the images based on the natural language text caption, and a confidence determination of whether FOD is present in the natural language text caption;

identifying, by an explainable AI model, words and phrases within the natural language text caption according to which the text classifier AI model made the classification and the confidence determination of whether FOD is present in the natural language text caption; and

displaying, in a user interface, a report including, for each identified FOD, a location where the FOD was identified in the inspection area, the captured image with the natural language text caption, and color coded highlighting of the key words and phrases identified by the explainable AI model to determine whether FOD is present in the natural language text caption.

17 . The computer program product of claim 16 , wherein the images are captured by:

receiving, from a tracking device, positional tracking data of a user moving through the defined inspection area; and

directing the number of imaging devices along a path through the inspection area defined by the positional tracking data to capture the images.

18 . The computer program product of claim 16 , further comprising instructions for translating, by a generative AI language translation model, the natural language explanations into a specified second language.

19 . The computer program product of claim 16 , wherein the imaging devices comprises at least one of:

Simultaneous Localization and Mapping devices;

drones; or

static cameras.

20 . The computer program product of claim 16 , wherein the captured images comprise at least one of:

still images; or

video.

21 . The computer program product of claim 16 , wherein the explainable AI model comprises at least one of:

a SHapley Additive explanations model;

a Local Interpretable Model Agnostic Explanation model;

tree surrogates;

Global Interpretation via Recursive Partitioning;

Explainable Boosting Machine; or

Contrastive Explanation Method.