System and method of operation for remotely operated vehicles for automatic detection of structure integrity threats
The present invention provides a system and method of automatic detection of structure integrity threats. A threat detection engine detects integrity threats in structures, such as underwater structures, and segments the structures in an image using convolutional neural networks (“CNN”). The threat detection engine may include a dataset module, a CNN training module, a segmentation map module, a semi-supervision module, and an efficiency module. The threat detection engine may train a deep learning model to detect anomalies in videos. To do so, a dataset module with videos may be used where the dataset module includes annotations detailing at what timestamps one or more anomalies are visible.
1. A system for operating a remotely operated vehicle (ROV) comprising:
a database module of 3D elements operable to represent objects disposed in an operation environment of the ROV;
a virtual video generating module operable to generate a virtual video incorporating the 3D elements;
a video camera mounted to the ROV operable to generate a real video of the operation environment of the ROV;
a synchronizing module operable to synchronize an angle and position of a virtual camera with an angle and position of the video camera mounted to the ROV;
a visualization engine operable to superimpose the real video on the virtual video to create hybrid 3D imagery; and
a threat detection engine operable to detect an integrity threat in a structure from the hybrid 3D imagery and segment the structure in the hybrid 3D imagery, the threat detection engine comprising:
a convolutional neural network (CNN) training module comprising a semi-supervised dataset and a supervised dataset, wherein the semi-supervised data includes at least a portion of the real video, and wherein the supervised dataset includes at least a portion of the real video and at least a portion of the virtual video;
a segmentation map module; and
an efficiency module.
2. The system of claim 1 , wherein the CNN training module includes a timestamp for an image with an anomaly from the real video.
3. The system of claim 2 , wherein the CNN training module analyzes the image and either outputs a logic high if the anomaly is detected or outputs a logic low otherwise.
4. The system of claim 3 , wherein the CNN training module comprises a plurality of stacked convolutional layers, wherein each subsequent stacked convolutional layer of the plurality of stacked convolutional layers includes a larger region of the input image.
5. The system of claim 4 , wherein the CNN training model further comprises a coarse structure segmentation map.
6. The system of claim 5 , wherein the segmentation map module generates a segmentation map dataset using pixel-level segmentations.
7. The system of claim 6 , wherein the segmentation map module generates the pixel-level segmentations by projecting a 3D model of a visible structure into the ROV's virtual camera.
8. The system of claim 7 , wherein the CNN training module trains a CNN model to minimize a loss function.
9. The system of claim 8 , wherein the CNN training module (i) uses a loss function L for data that contains both segmentation data and anomaly ground-truth data and (ii) uses a loss function La for data that contains anomaly ground-truth data but not segmentation data.
10. The system of claim 9 , wherein the efficiency module computes a binary mask m and, when all of m's elements are close to zero, the efficiency module stops the threat detection engine from making further computations and generate an output that there are not structural anomalies.
11. A system for undersea exploration comprising:
a remote operated vehicle (ROV) comprising a camera for acquiring a real video;
a networked operating system comprising a computer and computer executable software comprising a visualization engine and a threat detection engine;
a database module of 3D elements operable to represent objects disposed in an operation environment of the ROV;
a virtual video generating module operable to generate a virtual video incorporating the 3D elements;
a video camera mounted to the ROV operable to generate a real video of the operation environment of the ROV;
a synchronizing module operable to synchronize an angle and position of a virtual camera with an angle and position of the video camera mounted to the ROV;
wherein the visualization engine is operable to superimpose the real video on the virtual video to create hybrid 3D imagery; and
wherein the threat detection engine is operable to detect an integrity threat in a structure from the hybrid 3D imagery and segment the structure in the hybrid 3D imagery, the threat detection engine comprising:
a convolutional neural network (CNN) training module comprising a semi-supervised dataset and a supervised dataset, wherein the semi-supervised data includes at least a portion of the real video, and wherein the supervised dataset includes at least a portion of the real video and at least a portion of the virtual video;
a segmentation map module;
an efficiency module; and
a navigation interface configured to display the hybrid 3D imagery, the navigation interface comprising at least one networked monitor.
12. The system of claim 11 , wherein CNN training module includes a timestamp for an image with an anomaly from the real video.
13. The system of claim 12 , wherein the CNN training module analyzes the image and outputs a logic high if a visible anomaly is detected and outputs a logic low otherwise.
14. The system of claim 13 , wherein the CNN training module comprises a plurality of stacked convolutional layers, wherein each subsequent stacked convolutional layer of the plurality of stacked convolutional layers includes a larger region of the input image.
15. The system of claim 14 , wherein the CNN training model further comprises a coarse structure segmentation map.
16. The system of claim 15 , wherein the segmentation map module generates a segmentation map dataset using pixel-level segmentations.
17. The system of claim 16 , wherein the segmentation map module generates the pixel-level segmentations by projecting a 3D model of a visible structure into the ROV's virtual camera.
18. A method of operating a remotely operated vehicle (ROV) comprising:
obtaining 3D data;
storing 3D elements in a database module, the 3D elements representing objects disposed in the ROV's operation environment and comprising the 3D data;
generating a virtual video of the 3D elements;
synchronizing an angle and position of a virtual camera with an angle and position of a video camera mounted to the ROV; and
aligning and superimposing a virtual video element with a real video element to create hybrid 3D imagery;
segmenting a structure from the hybrid 3D imagery;
training a CNN model with a semi-supervised dataset and a supervised dataset, wherein the semi-supervised data includes at least a portion of the real video, and wherein the supervised dataset includes at least a portion of the real video and at least a portion of the virtual video; and
detecting an integrity threat in the structure from the hybrid 3D imagery.
19. The method of claim 18 , wherein detecting an integrity threat further includes
generating segmentation maps.
20. The method of claim 19 , wherein detecting an integrity threat further includes:
generating a segmentation map dataset using pixel-level segmentations;
computing a binary mask m; and
stopping further computations when all of m's elements are close to zero.