IP Library Granted Patent US 11,755,643
Granted Patent B2
US 11,755,643 · App. 16/921,248 · Granted Sep 12, 2023

Metadata generation for video indexing

Inventors: Zvi Figov (Modiin, IL); Irit Ofer (Kfar Saba, IL)
Assignee: Microsoft Technology Licensing, LLC
G06F16/71G06F16/7837G06V10/25G06V20/41G06V20/48
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,755,643
App. No.
16/921,248
Granted
Sep 12, 2023
Kind
B2
Abstract

A video indexing system identifies groups of frames within a video frame sequence captured by a static camera during a same scene. Context metadata is generated for each frame in each group based on an analysis of fewer than all frames in the group. The frames are indexed in a database in association with the generated context metadata.

Claims (55)

1. A method comprising:

analyzing a sequence of video frames to determine whether the sequence was captured by a static camera;

responsive to determining that the sequence of frames was captured by the static camera, subjecting the sequence of frames to a first series of processing operations that includes generating context metadata for each frame in the sequence based on an analysis of fewer than all frames in the sequence; and

responsive to the determining that a select frame of the sequence of frames was captured by a moving camera, subjecting the select frame to a second series of processing operations for generating the context metadata that is different than the first series of processing operations.

2. The method of claim 1 , wherein the second series of processing operations generates context metadata for the select frame based on an image analysis that is limited to the select frame.

3. The method of claim 1 , wherein generating context metadata for each frame in the sequence based on the analysis of fewer than all frames in the sequence further comprises:

selecting a keyframe from the sequence;

generating at least a portion of the context metadata based on an analysis of the keyframe without analyzing other frames of the sequence; and

indexing the other frames of the sequence in association with the generated context metadata.

4. The method of claim 3 , wherein generating the context metadata further comprises:

generating descriptors for multiple objects present in the keyframe.

5. The method of claim 4 , wherein generating the context metadata further comprises:

generating a scene label generated based on the descriptors generated for the keyframe.

6. The method of claim 3 , wherein generating the context metadata further comprises:

generating a region of interest (ROI) mask for the keyframe;

applying the ROI mask to each of the other frames in the sequence; and

omitting an area defined by the ROI mask from subsequent processing operations performed on each of the other frames of the sequence.

7. The method of claim 1 , further comprising:

responsive to determining that the sequence of frames was captured by a static camera, determining a size of an object present in multiple frames of the sequence by analyzing a position of the object relative to a fixed reference point that appears within each of the multiple frames.

8. The method of claim 1 , further comprising:

responsive to determining that the sequence of frames was captured by the static camera, executing object tracking logic that assumes a fixed camera frame of reference.

9. A video indexing system comprising:

a frame classifier that classifies video frames received as part of a sequence into different static camera scene groups;

a context metadata generation engine that:

receives a group of frames classified as comprising a same static camera scene group of the different static camera scene groups; and

analyzes fewer than all frames in the group to generate context metadata for each frame in the group; and

an indexing engine that indexes each frame in the group in a database in association with the generated context metadata.

10. The video indexing system of claim 9 , wherein the context metadata generation engine is further adapted to:

determine that a select frame of the sequence was captured by a moving camera, based on a classification of the frame classifier; and

responsive to the determination, generate context metadata for the select frame based on an image analysis limited to the select frame.

11. The video indexing system of claim 9 , wherein the context metadata generation engine is further adapted to:

select a keyframe from the group; and

generate at least a portion of the context metadata for each frame in the group based on an analysis of the keyframe without analyzing other frames of the group.

12. The video indexing system of claim 11 , wherein the context metadata generation engine is adapted to:

generate a region of interest (ROI) mask for the keyframe of the group;

apply the ROI mask to each frame in the group; and

omit an area defined by the ROI mask from subsequent processing operations performed on the frames of the group.

13. The video indexing system of claim 11 , wherein the context metadata includes descriptors for multiple objects present in the keyframe.

14. The video indexing system of claim 13 , wherein the context metadata includes a scene label generated based on the descriptors generated for the keyframe.

15. The video indexing system of claim 9 , further comprising:

an object tracker that executes logic to track movements of objects throughout multiple frames of the group, the logic being adapted to implement constraints that assume a fixed frame of reference for a camera capturing the video.

16. One or more tangible computer-readable storage media encoding computer-executable instructions for executing a computer processes comprising:

classifying video frames within a sequence into different static camera scene groups;

generating context metadata for each frame in a group of the different static camera scene groups, the context metadata being generated based on an analysis of fewer than all frames in the group; and

indexing each frame in the group in a database in association with the generated context metadata.

17. The one or more tangible computer-readable storage media of claim 16 , wherein generating the context metadata further comprises:

selecting a keyframe from the group; and

generating at least a portion of the context metadata for each frame in the group based on an analysis of the keyframe without analyzing other frames of the group.

18. The one or more tangible computer-readable storage media of claim 16 , wherein the computer process further comprises:

determining that a select frame of the sequence was captured by a moving camera;

responsive to the determination, generating context metadata for the select frame based on an image analysis limited to the select frame.

19. The one or more tangible computer-readable storage media of claim 17 , wherein the computer process further comprises:

generating a region of interest (ROI) mask for the keyframe of the group;

applying the ROI mask to each frame in the group; and

omitting an area defined by the ROI mask from subsequent processing operations performed on the frames of the group.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 6, 2020
From: FIGOV, ZVI; OFER, IRIT
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 053126/0693 →
Continuity (1)
Related Publication 20220004574A1 · Jan 6, 2022