Anomaly tracking system and method using enterprise digital twins based mixed reality
A system and method for 3D anomaly detection and tracking are provided that uses multimodal fusion, reduced training data, recursive segmentation and histogram statistic distance. The anomaly may be a defect or a configuration error.
1 . A method for anomaly detection, the method comprising:
tracking, on a backend computer system having a processor, an anomaly of an actual object in a 3D scene to generate a mixed reality environment that visualizes the anomaly of the actual object in the mixed reality environment;
training, by the processor, at least two deep learning models using at least one benchmark dataset to generate at least two sets of labels for one or more objects in the 3D scene;
performing, by the processor, recursive cluster segmentation of the one or more objects in the 3D scene to generate at least two sets of histograms of cluster labels for the at least two trained deep learning models;
merging, by the processor, the sets of histograms of cluster labels for the at least two trained deep learning models for each cluster to generate a vector for each cluster;
training, by the processor, a machine learning model based on the merged sets of labels from the at least two trained deep learning models which reduce a complexity of a point cloud of the 3D scene by representing raw RGB and XYZ data of the point cloud in a histogram/distribution of labels of each 3D point; and
performing, by the processor using the machine learning model, inference to identify the anomaly in an object of the one or more objects in the 3D scene.
2 . The method of claim 1 , wherein performing the inference further comprises identifying the anomaly in the object by one of identifying a digital twin and determining if a defect resolution is met.
3 . The method of claim 1 further comprising labeling the identified anomaly for visualization.
4 . The method of claim 3 , wherein labeling the identified anomaly further comprising coloring the identified anomaly.
5 . The method of claim 4 further comprising visualizing, on a mixed reality device, the colored identified anomaly.
6 . The method of claim 1 , wherein training, by the processor, the at least two deep learning models further comprises training, by the processor, a PointCNN model using scannet benchmark data to generate a first set of labels for the 3D scene, training, by the processor, a second PointCNN model using shapenet benchmark data to generate a second set of labels for the 3D scene, training, by the processor, a RandLA model using S3DIS benchmark data to generate a third set of labels for the 3D scene, training, by the processor, a 3DBonet model using the S3DIS benchmark data to generate a fourth set of labels for the 3D scene and training, by the processor, a second 3DBonet model using a few shot training data to generate a fifth set of labels for the 3D scene.
7 . The method of claim 6 , wherein performing the recursive cluster segmentation further comprises performing, by the processor, recursive cluster segmentation on each set of labels to generate a histogram for each set of labels.
8 . The method of claim 1 , wherein the anomaly is one of a manufacturing defect, a material defect and a configuration error.
9 . A system, comprising:
a computer system having a processor that executes a plurality of lines of computer code, the computer system being configured to:
track an anomaly of an actual object in a 3D scene to generate a mixed reality environment that visualizes the anomaly of the actual object in the mixed reality environment;
train at least two deep learning models using at least one benchmark dataset to generate at least two sets of labels for one or more objects in the 3D scene;
perform recursive cluster segmentation of the one or more objects in the 3D scene to generate at least two sets of histograms of cluster labels for the at least two trained deep learning models;
merge the sets of histograms of cluster labels for the at least two trained deep learning models for each cluster to generate a vector for each cluster;
train a machine learning model based on the merged sets of labels from the at least two trained deep learning models which reduce a complexity of a point cloud of the 3D scene by representing raw RGB and XYZ data of the point cloud in a histogram/distribution of labels of each 3D point; and
perform, using the machine learning model, inference to identify the anomaly in an object of the one or more objects in the 3D scene.
10 . The system of claim 9 , wherein the computer system is further configured to perform the inference by one of the identification of a digital twin and the determination if a defect resolution is met.
11 . The system of claim 9 , wherein the computer system is further configured to label the identified anomaly for visualization.
12 . The system of claim 11 , wherein the computer system is further configured to color the identified anomaly.
13 . The system of claim 12 further comprising a mixed reality device connected to the computer system that visualizes the colored identified anomaly.
14 . The system of claim 13 , wherein the mixed reality device is a mixed reality headset.
15 . The system of claim 9 , wherein the computer system that trains the at least two deep learning models is further configured to train a PointCNN model using scannet benchmark data to generate a first set of labels for the 3D scene, train a second PointCNN model using shapenet benchmark data to generate a second set of labels for the 3D scene, train a RandLA model using S3DIS benchmark data to generate a third set of labels for the 3D scene, train a 3DBonet model using the S3DIS benchmark data to generate a fourth set of labels for the 3D scene and train a second 3DBonet model using a few shot training data to generate a fifth set of labels for the 3D scene.
16 . The system of claim 9 , wherein the anomaly is one of a manufacturing defect, a material defect and a configuration error.