Context processor for video analysis system
View Patent ↗Embodiments of the present invention provide a method and a system for mapping a scene depicted in an acquired stream of video frames that may be used by a machine-learning behavior-recognition system. A background image of the scene is segmented into plurality of regions representing various objects of the background image. Statistically similar regions may be merged and associated. The regions are analyzed to determine their z-depth order in relation to a video capturing device providing the stream of the video frames and other regions, using occlusions between the regions and data about foreground objects in the scene. An annotated map describing the identified regions and their properties is created and updated.
1. A method for mapping a scene depicted in a sequence of video frames, the method comprising:
segmenting a background image representing static elements of the scene into one or more spatially ordered regions, wherein each spatially ordered region has a corresponding a set of pixels in the background image;
receiving a current video frame, of the sequence;
identifying one or more foreground objects depicted in the current video, wherein each foreground object occludes the pixels in one or more of the spatially ordered regions of the background image;
analyzing the one or more spatially ordered regions of the background image to identify at least a spatial ordering for one or more of the foreground objects, relative to the spatial ordering of the one or more spatially ordered regions of the background image; and
generating, based on the spatially ordered regions of the scene and the spatial ordering for the one or more foreground objects, an annotated map of the scene specifying scene geometry for one or more of the spatially ordered regions of the scene.
2. The method of claim 1 , further comprising, updating, based on the current video frame, the pixels in at least a first one of the spatially ordered regions of the background image.
3. The method of claim 1 , wherein each foreground object is defined by a collection of pixels of the current frame having one or more features different from a corresponding set of pixels in the background image.
4. The method of claim 1 , further comprising:
determining a first physical characteristic of at least a first one of the spatially ordered regions, wherein the first physical characteristic is one of position, orientation, size, shape, class, and distance scale.
5. The method of claim 1 , wherein analyzing the one or more spatially ordered regions of the background image to identify at least a spatial ordering for the one or more foreground objects, relative to the spatial ordering of the one or more spatially ordered regions of the background image comprises determining that at least a portion of a first one of the spatially ordered regions of the background image occludes at least a portion of the foreground object and that at least a portion of a second one of the spatially ordered regions of the background image is occluded by the foreground object.
6. A computer-readable storage medium containing a program, which, when executed on a processor is configured to perform an operation for mapping a scene depicted in a sequence of video frames, the operation comprising:
segmenting a background image representing static elements of the scene into one or more spatially ordered regions, wherein each spatially ordered region has a corresponding a set of pixels in the background image;
receiving a current video frame, of the sequence;
identifying one or more foreground objects depicted in the current video, wherein each foreground object occludes the pixels in one or more of the spatially ordered regions of the background image;
analyzing the one or more spatially ordered regions of the background image to identify at least a spatial ordering for one or more of the foreground objects, relative to the spatial ordering of the one or more spatially ordered regions of the background image; and
generating, based on the spatially ordered regions of the background image and the spatial ordering for the one or more foreground objects, an annotated map of the scene specifying scene geometry for one or more of the plurality of regions in the background image.
7. The computer-readable storage medium of claim 6 , wherein the operation further comprises, updating, based on the current video frame, the pixels in at least a first one of the spatially ordered regions of the background image.
8. The computer-readable storage medium of claim 6 , wherein each foreground object is defined by a collection of pixels of the current frame having one or more features different from a corresponding set of pixels in the background image.
9. The computer-readable storage medium of claim 6 , wherein the operation further comprises:
determining a first physical characteristic of at least a first one of the spatially ordered regions, wherein the first physical characteristic is one of position, orientation, size, shape, class, and distance scale.
10. The computer-readable storage medium of claim 6 , wherein analyzing the one or more spatially ordered regions of the background image to identify at least a spatial ordering for the one or more foreground objects, relative to the spatial ordering of the one or more spatially ordered regions of the background image comprises determining that at least a portion of a first one of the spatially ordered regions of the background image occludes at least a portion of the foreground object and that at least a portion of a second one of the spatially ordered regions of the background image is occluded by the foreground object.
11. A system comprising,
a processor; and
a memory containing a context processor application, which when executed by the processor is configured to perform an operation for mapping a scene depicted in a sequence of video frames, the operation comprising:
segmenting a background image representing static elements of the scene into one or more spatially ordered regions, wherein each spatially ordered region has a corresponding a set of pixels in the background image,
receiving a current video frame, of the sequence,
identifying one or more foreground objects depicted in the current video, wherein each foreground object occludes the pixels in one or more of the spatially ordered regions of the background image,
analyzing the one or more spatially ordered regions of the background image to identify at least a spatial ordering for one or more of the foreground objects, relative to the spatial ordering of the one or more spatially ordered regions of the background image, and
generating, based on the spatially ordered regions of the background image and the spatial ordering for the one or more foreground objects, an annotated map of the scene specifying scene geometry for one or more of the plurality of regions in the background image.
12. The system of claim 11 , wherein the operation further comprises, updating, based on the current video frame, the pixels in at least a first one of the spatially ordered regions of the background image.
13. The system of claim 11 , wherein each foreground object is defined by a collection of pixels of the current frame having one or more features different from a corresponding set of pixels in the background image.
14. The system of claim 11 , wherein the operation further comprises:
determining a first physical characteristic of at least a first one of the spatially ordered regions, wherein the first physical characteristic is one of position, orientation, size, shape, class, and distance scale.
15. The system of claim 11 , wherein analyzing the one or more spatially ordered regions of the background image to identify at least a spatial ordering for the one or more foreground objects, relative to the spatial ordering of the one or more spatially ordered regions of the background image comprises determining that at least a portion of a first one of the spatially ordered regions of the background image occludes at least a portion of the foreground object and that at least a portion of a second one of the spatially ordered regions of the background image is occluded by the foreground object.