IP Library › Granted Patent US 12670403

Granted Patent B2

US 12670403 · App. 17/453,818 · Granted Jun 30, 2026

Classifier with outlier detection algorithm

Inventors: Anapathur Viswanathan Ramesh (Bothell, WA); Michael Thomas Swayne (Redmond, WA); Huafeng Yu (Madison, AL)

Assignee: The Boeing Company

G06N3/088 G06F18/24 G06N3/045 G06V30/248 G06V30/2528

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12670403
App. No.: 17/453,818
Granted: Jun 30, 2026
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

A classifier is executed including an unsupervised artificial intelligence model and a supervised artificial intelligence model. The classifier is configured to receive run-time input data, and process the run-time input data using the unsupervised artificial intelligence model and an outlier detection algorithm to determine whether the run-time input data is an outlier as compared to training input data. Responsive to determining that the run-time input data is not an outlier, the classifier determines a predicted response label for the run-time input based on the run-time input data processed using the supervised artificial intelligence model. Responsive to determining that the run-time input data is an outlier, the classifier refrains from determining the predicted response label for the run-time input based on the run-time input data processed using the supervised artificial intelligence model, and instead outputs a prompt for user input of a user-curated response label for the run-time input.

Claims (87)

1 . A computing system, comprising:

a processor and a non-volatile memory storing executable instructions that, in response to execution by the processor, cause the processor to:

execute a classifier including an unsupervised artificial intelligence model and a supervised artificial intelligence model, the supervised artificial intelligence model being logically distinct from the unsupervised artificial intelligence model, the unsupervised artificial intelligence model comprising a first neural network and the supervised artificial intelligence model comprising a second neural network, the classifier being configured to:

receive run-time input data;

process the run-time input data using the unsupervised artificial intelligence model and an outlier detection algorithm to determine whether the run-time input data is an outlier as compared to training input data;

responsive to determining that the run-time input data is an outlier, refrain from determining the predicted response label for the run-time input data based on the run-time input data processed using the supervised artificial intelligence model, and instead output a prompt for user input of a user-curated response label for the run-time input data,

wherein the unsupervised artificial intelligence model includes a clustering algorithm or a self-organizing map configured to identify a plurality of clusters in the training input data, classify the run-time input data as belonging to a classified cluster of the plurality of identified clusters, wherein the outlier detection algorithm is configured to determine whether the run-time input data is an outlier based upon a proximity of the run- time input data to the classified cluster; and

wherein the clustering algorithm implements one or more of hierarchical clustering, k-means clustering, mixture modeling, a DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm, and an OPTICS (Ordering Points To Identify the Clustering Structure) algorithm.

2 . The computing system of claim 1 ,

wherein the first neural network has been trained on the training input data;

wherein the second neural network has been trained on the training input data and ground truth labels that are associated with the training input data;

wherein the classifier is configured to process the run-time input data using the unsupervised artificial intelligence model at least in part by extracting a first set of features from the run-time input data using the unsupervised artificial intelligence model; and

wherein the classifier is configured to process the run-time input data using the supervised artificial intelligence model at least in part by extracting a second set of features from the run-time input data using the supervised artificial intelligence model.

3 . The computing system of claim 2 ,

wherein the training input data includes a collection of images from a camera on board a vehicle, the images capturing an external environment of the vehicle;

wherein the ground truth labels associated with the training input data include a vehicle type label for each of the images selected from a defined set of vehicle type labels; and

wherein the predicted response label is one of the vehicle type labels.

4 . The computing system of claim 3 , wherein the vehicle is an aircraft and the vehicle type labels are aircraft type labels.

5 . The computing system of claim 1 , wherein the processor is further configured to, when the run-time input data is determined to be an outlier:

receive the user input including the user-curated response label for the training input data;

pair the user-curated response label with the run-time input data to create a feedback training data pair;

perform feedback training of the supervised artificial intelligence model based on the run-time input data and the user-curated response label of the feedback training data pair;

perform feedback training of the unsupervised artificial intelligence model based on the run-time input data; and

output an alert indicating that the outlier has been detected.

6 . The computing system of claim 1 , wherein the processor is further configured to, when the run-time input data is determined not to be an outlier:

pair the predicted response label with the run-time input data to create a feedback training data pair;

perform feedback training of the supervised artificial intelligence model based on the run-time input data and the predicted response label of the feedback training data pair; and

perform feedback training of the unsupervised artificial intelligence model based on the run-time input data.

7 . The computing system of claim 1 , wherein the first neural network includes an input layer connected to one or more convolutional layers configured to extract features from the run-time input data, and an output layer configured to output a cluster identification and a proximity to a cluster for the run-time input data.

8 . The computing system of claim 1 , wherein the supervised artificial intelligence model includes an input layer connected to one or more convolutional layers, additionally includes one or more fully connected deep learning layers connected to the convolutional layers, and further includes an output layer including a plurality of nodes each indicating a respective predicted response label.

9 . The computing system of claim 1 ,

wherein the run-time input data is sensor data;

wherein the unsupervised artificial intelligence model and the supervised artificial intelligence model are LSTM (long short-term memory) recurrent neural networks; and

wherein the predicted response label is a time regression.

10 . A method for a computing system comprising a processor and a non-volatile memory storing executable instructions, the method comprising:

executing a classifier including an unsupervised artificial intelligence model and a supervised artificial intelligence model, the supervised artificial intelligence model being logically distinct from the unsupervised artificial intelligence model, the unsupervised artificial intelligence model comprising a first neural network and the supervised artificial intelligence model comprising a second neural network, executing the classifier including:

receiving run-time input data;

processing the run-time input data using the unsupervised artificial intelligence model and an outlier detection algorithm to determine whether the run-time input data is an outlier as compared to training input data;

responsive to determining that the run-time input data is not an outlier, determining a predicted response label for the run-time input data based on the run-time input data processed using the supervised artificial intelligence model; and

responsive to determining that the run-time input data is an outlier, refraining from determining the predicted response label for the run-time input data based on the run-time input data processed using the supervised artificial intelligence model, and instead outputting a prompt for user input of a user-curated response label for the run-time input data,

wherein the clustering algorithm of the unsupervised artificial intelligence model implements one or more of hierarchical clustering, k-means clustering, mixture modeling, a DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm, and an OPTICS (Ordering Points To Identify the Clustering Structure) algorithm.

11 . The method of claim 10 ,

wherein the first neural network has been trained on the training input data; and

wherein the second neural network has been trained on the training input data and ground truth labels that are associated with the training input data;

12 . The method of claim 11 ,

wherein the training input data includes a collection of images from an on-board camera, the images capturing an external environment of a vehicle;

wherein the ground truth labels associated with the training input data include a vehicle type label for each of the images selected from a defined set of vehicle type labels; and

wherein the predicted response label is one of the vehicle type labels.

13 . The method of claim 12 , wherein the vehicle is an aircraft and the vehicle type labels are aircraft type labels.

14 . The method of claim 10 , further comprising, when the run-time input data is determined to be an outlier:

receiving the user input including the user-curated response label for the training input data;

pairing the user-curated response label with the run-time input data to create a feedback training data pair;

performing feedback training of the supervised artificial intelligence model based on the run-time input data and the user-curated response label of the feedback training data pair;

performing feedback training of the unsupervised artificial intelligence model based on the run-time input data; and

outputting an alert indicating that the outlier has been detected, and

when the run-time input data is determined not to be an outlier:

pairing the predicted response label with the run-time input data to create a feedback training data pair;

performing feedback training of the supervised artificial intelligence model based on the run-time input data and the predicted response label of the feedback training data pair; and

performing feedback training of the unsupervised artificial intelligence model based on the run-time input data.

15 . The method of claim 10 , wherein the first neural network includes an input layer connected to one or more convolutional layers configured to extract features from a run-time input vector, and an output layer configured to output a cluster identification and a proximity to a cluster for the run-time input data.

16 . The method of claim 10 , wherein the supervised artificial intelligence model includes an input layer connected to one or more convolutional layers, additionally includes one or more fully connected deep learning layers connected to the convolutional layers, and further includes an output layer including a plurality of nodes each indicating a respective predicted response label.

17 . The method of claim 10 ,

wherein the run-time input data is sensor data;

wherein the unsupervised artificial intelligence model and the supervised artificial intelligence model are LSTM (long short-term memory) recurrent neural networks; and

wherein the predicted response label is a time regression.

18 . A computing system, comprising:

a processor and a non-volatile memory storing executable instructions that, in response to execution by the processor, cause the processor to:

receive run-time input data including an image;

responsive to determining that the run-time input data is not an outlier, determine a predicted response label for the run-time input data based on the run- time input data processed using the supervised artificial intelligence model, wherein the predicted response label is one vehicle type label from a defined set of vehicle type labels; and

wherein the training input data includes a collection of images from a camera on board a vehicle, the images capturing an external environment of the vehicle;

wherein ground truth labels associated with the training input data include a vehicle type label for each of the images selected from the defined set of vehicle type labels; and

wherein the vehicle is an aircraft and the vehicle type labels are aircraft type labels.

19 . The computing system of claim 18 , wherein the clustering algorithm implements one or more of hierarchical clustering, k-means clustering, mixture modeling, a DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm, and an OPTICS (Ordering Points To Identify the Clustering Structure) algorithm.

20 . The computing system of claim 18 , wherein the processor is further configured to, when the run-time input data is determined to be an outlier:

receive the user input including the user-curated response label for the training input data;

pair the user-curated response label with the run-time input data to create a feedback training data pair;

perform feedback training of the supervised artificial intelligence model based on the run-time input data and the user-curated response label of the feedback training data pair;

perform feedback training of the unsupervised artificial intelligence model based on the run-time input data; and

output an alert indicating that the outlier has been detected.