Detecting anamolous sea-surface oil based on a synthetic discriminant signal and learned patterns of behavior
View Patent ↗A behavioral recognition system may include both a computer vision engine and a machine learning engine configured to observe and learn patterns of behavior in video data. Certain embodiments may be configured to detect and evaluate the presence of sea-surface oil on the water surrounding an offshore oil platform. The computer vision engine may be configured to segment image data into detected patches or blobs of surface oil (foreground) present in the field of view of an infrared camera (or cameras). A machine learning engine may evaluate the detected patches of surface oil to learn to distinguish between sea-surface oil incident to the operation of an offshore platform and the appearance of surface oil that should be investigated by platform personnel.
1. A computer-implemented method for analyzing a scene depicted in a plurality of input streams of video frames, the method comprising:
multiplexing the plurality of input streams of video frames, based on differences between the input streams corresponding to contrasts between wavelength ranges in which difference between radiance from seawater is substantially greater and/or less than difference between radiance from oil, to generate a combined stream of video frames, wherein each of the plurality of input streams of video frames capture the scene in a respective infrared (IR) spectral band, and wherein each pixel in the combined stream of video frames has a value indicating closeness of a match between an IR signature of the pixel and an IR signature of oil;
learning expected patterns of behavior in the scene based on the combined stream of video frames;
detecting oil observed in one or more subsequent video frames in the combined stream of video frames that does not correspond to the learned expected patterns; and
displaying a report indicating the detected oil that does not correspond to the learned expected patterns.
2. The method of claim 1 , wherein the multiplexing the plurality of input streams of video frames includes:
determining differences between pairs of input streams in the plurality of input streams of video frames; and
determining the values of the pixels in the combined stream of video frames based on the determined differences between the pairs of input streams, wherein the determined pixel values are directly proportional to one or more of the determined differences which correspond to contrasts between wavelength ranges in which difference between radiance from seawater is substantially greater than difference between radiance from oil, and inversely proportional to one or more of the determined differences which correspond to contrasts between wavelength ranges in which difference between radiance from oil is substantially greater than difference between radiance from seawater, or vice versa.
3. The method of claim 2 , wherein the determined values of the pixels in the combined stream of video frames are normalized to a range, one end of which represents water and another end of which represents oil.
4. The method of claim 1 , wherein the plurality of input streams of video frames are captured by three or more long-wavelength infrared (LWIR) cameras whose signals are filtered by band-pass filters allowing light in respective wavelength ranges to pass.
5. The method of claim 1 , wherein the plurality of input streams of video frames are captured by one or more spectrometers.
6. The method of claim 1 , wherein the plurality of input streams of video frames capture the scene of an area surrounding an offshore oil platform.
7. The method of claim 1 , wherein the plurality of input streams of video frames capture the scene in respective wavelength ranges of about 8-9 micrometers, 8-11.5 micrometers, and 8-13 micrometers.
8. The method of claim 1 , wherein learning the expected patterns of behavior in the scene includes:
separating foreground blobs depicting oil from background depicting water based on the values of the pixels in the video frames in the combined stream of video frames and one or more features associated with the foreground blobs, wherein the features include at least one of locations of the blobs, sizes of the blobs, rates of change in blob sizes, a measure of intensity of the blobs, motion characteristics of the blobs, whether the blobs have non-sharp edges, whether the blobs have high fractal dimension, and whether the blobs are asymmetrical; and
providing the foreground blobs and associated features to a machine learning engine that autonomously builds a model of the expected patterns of behavior in the scene.
9. The method of claim 8 , wherein the machine learning engine includes one of an adaptive resonance theory (ART) network or a combination of an ART network and a self-organizing map (SOM).
10. A non-transitory computer-readable storage medium storing instructions, which when executed by a computer system, perform operations for analyzing a scene depicted in a plurality of input streams of video frames, the operations comprising:
multiplexing the plurality of input streams of video frames, based on differences between the input streams corresponding to contrasts between wavelength ranges in which difference between radiance from seawater is substantially greater and/or less than difference between radiance from oil, to generate a combined stream of video frames, wherein each of the plurality of input streams of video frames capture the scene in a respective infrared (IR) spectral band, and wherein each pixel in the combined stream of video frames has a value indicating closeness of a match between an IR signature of the pixel and an IR signature of oil;
learning expected patterns of behavior in the scene based on video frames in the combined stream of video frames;
detecting oil observed in one or more subsequent video frames in the combined stream of video frames that does not correspond to the learned expected patterns; and
displaying a report indicating the detected oil that does not correspond to the learned expected patterns.
11. The computer-readable storage medium of claim 10 , wherein the multiplexing the plurality of input streams of video frames includes:
determining differences between pairs of input streams in the plurality of input streams of video frames; and
determining the values of the pixels in the combined stream of video frames based on the determined differences between the pairs of input streams, wherein the determined pixel values are directly proportional to one or more of the determined differences which correspond to contrasts between wavelength ranges in which difference between radiance from seawater is substantially greater than difference between radiance from oil, and inversely proportional to one or more of the determined differences which correspond to contrasts between wavelength ranges in which difference between radiance from oil is substantially greater than difference between radiance from seawater, or vice versa.
12. The computer-readable storage medium of claim 11 , wherein the determined values of the pixels in the combined stream of video frames are normalized to a range, one end of which represents water and another end of which represents oil.
13. The computer-readable storage medium of claim 10 , wherein the plurality of input streams of video frames are captured by three or more long-wavelength infrared (LWIR) cameras whose signals are filtered by band-pass filters allowing light in respective wavelength ranges to pass.
14. The computer-readable storage medium of claim 10 , wherein the plurality of input streams of video frames are captured by one or more spectrometers.
15. The computer-readable storage medium of claim 10 , wherein the plurality of input streams of video frames capture the scene of an area surrounding an offshore oil platform.
16. The computer-readable storage medium of claim 10 , wherein the plurality of input streams of video frames capture the scene in respective wavelength ranges of about 8-9 micrometers, 8-11.5 micrometers, and 8-13 micrometers.
17. The computer-readable storage medium of claim 10 , wherein learning the expected patterns of behavior in the scene includes:
separating foreground blobs depicting oil from background depicting water based on the values of the pixels in the video frames in the combined stream of video frames and one or more features associated with the foreground blobs, wherein the features include at least one of locations of the blobs, sizes of the blobs, rates of change in blob sizes, a measure of intensity of the blobs, motion characteristics of the blobs, whether the blobs have non-sharp edges, whether the blobs have high fractal dimension, and whether the blobs are asymmetrical; and
providing the foreground blobs and associated features to a machine learning engine that autonomously builds a model of the expected patterns of behavior in the scene.
18. The computer-readable storage medium of claim 17 , wherein the machine learning engine includes one of an adaptive resonance theory (ART) network or a combination of an ART network and a self-organizing map (SOM).
19. A system, comprising:
a processor; and
a memory, wherein the memory includes an application program configured to perform operations for analyzing a scene depicted in a plurality of input streams of video frames, the operations comprising:
multiplexing the plurality of input streams of video frames, based on differences between the input streams corresponding to contrasts between wavelength ranges in which difference between radiance from seawater is substantially greater and/or less than difference between radiance from oil, to generate a combined stream of video frames, wherein each of the plurality of input streams of video frames capture the scene in a respective infrared (IR) spectral band, and wherein each pixel in the combined stream of video frames has a value indicating closeness of a match between an IR signature of the pixel and an IR signature of oil,
learning expected patterns of behavior in the scene based on video frames in the combined stream of video frames,
detecting oil observed in one or more subsequent video frames in the combined stream of video frames that does not correspond to the learned expected patterns, and
displaying a report indicating the detected oil that does not correspond to the learned expected patterns.
20. The system of claim 19 , wherein the multiplexing the plurality of input streams of video frames includes:
determining differences between pairs of input streams in the plurality of input streams of video frames; and
determining the values of the pixels in the combined stream of video frames based on the determined differences between the pairs of input streams, wherein the determined pixel values are directly proportional to one or more of the determined differences which correspond to contrasts between wavelength ranges in which difference between radiance from seawater is substantially greater than difference between radiance from oil, and inversely proportional to one or more of the determined differences which correspond to contrasts between wavelength ranges in which difference between radiance from oil is substantially greater than difference between radiance from seawater, or vice versa.