Method and apparatus for multi-dimensional content search and video identification
View Patent ↗A multi-dimensional database and indexes and operations on the multi-dimensional database are described which include video search applications or other similar sequence or structure searches. Traversal indexes utilize highly discriminative information about images and video sequences or about object shapes. Global and local signatures around keypoints are used for compact and robust retrieval and discriminative information content of images or video sequences of interest. For other objects or structures relevant signature of pattern or structure are used for traversal indexes. Traversal indexes are stored in leaf nodes along with distance measures and occurrence of similar images in the database. During a sequence query, correlation scores are calculated for single frame, for frame sequence, and video clips, or for other objects or structures.
1. A method of organization of a multi-dimensional video database using a robust hash of a multi-dimensional vector signature as a traversal index, the method comprising:
generation of a robust hash value as a traversal index from multiple parameters extracted from a region of interest in a frame of a video sequence; and
storing data associated with the video sequence at a leaf node addressed by the robust hash value, wherein the leaf node is a member of a plurality of leaf nodes in a multi-dimensional video database.
2. The method of claim 1 further comprising:
using as the traversal index a combination of i) each keypoint of an image's global shape signature and ii) a motion description around the regions around the keypoint, and iii) global signatures of the image.
3. The method of claim 2 further comprising:
computing a set of histograms to generate a shape signature while using the selected keypoint, as a reference for the shape signature; and
creating a weighted sum of features extracted at points on the contour shape in each sector of multiple rings around the keypoint and having a reference orientation around a selected axis.
4. The method of claim 1 where the traversal index of the multi-dimensional video database uses pivot values around cluster centers.
5. The method of claim 1 where the database is generated from multiple views and aspects of an object.
6. The method of claim 1 wherein the traversal index is generated from a shape signature of a region around a selected object in the frame of the video sequence.
7. The method of claim 6 further comprising:
computing a set of histograms to generate the shape signature while using a center point as a point on a contour shape; and
creating a weighted sum of features extracted at points on the contour shape in each sector of multiple rings around the center point and having a reference orientation around a selected axis.
8. The method of claim 1 wherein the generation of the traversal index comprises:
generating the traversal index from multiple parameters extracted from an object or object bounding boxes in the frame of the video sequence.
9. The method of claim 1 , wherein a region around a selected region of interest is used to generate the traversal index and the leaf node stores a location and size of the region.
10. The method of claim 1 wherein the traversal index is a multi-dimensional vector comprising a single bit or a few bits per each dimension of the multi-dimensional vector.
11. The method of claim 1 wherein the traversal index uses signatures generated from a unique video sequence or other multi-dimensional content, the signatures based on content in a plurality of selected frames.
12. The method of claim 1 wherein frames are selected using a descriptive parameter of the video sequence including a sum of mean absolute difference (MAD) between frames and selecting the descriptive parameter minima or maxima.
13. The method of claim 1 wherein signatures from two selected frames are combined and used as a single combined signature, wherein the single combined signature reduces the correlation cost of storage and processing required for a large set of possible video sequences.
14. The method of claim 1 wherein a signature is centered around a region of interest and is based on shape, gradient and motion feature histograms in sectors and bins.
15. The method of claim of 1 wherein a signature is generated based on a set of local features around a region of interest.
16. The method of claim 15 wherein the local features include a set of features formed within the region of interest and second order gradient features within the region of interest.
17. The method of claim of 15 wherein the signature is generated based on local features around the region of interest as well as global features around the region of interest, wherein the global features include texture and color of an image.
18. The method of claim 15 wherein the local features comprise first and second order gradient features.
19. The method of claim of 1 and wherein a signature is generated based on local features around a region of interest, wherein the local features comprise shape features within the region of interest and the local features based on motion direction.
20. A method to post process signatures and associated data to evaluate a video sequence of interest and stored video sequences to increase accuracy and confidence of a video sequence match, the method comprising:
calculating a distance between signatures of a query and signatures of features of a video database entry including signatures for a region around a keypoint to identify candidate matching frames;
correlating changes in signatures between the query and the video database entry for the candidate matching frames to provide a factor in a sequence correlation score to identify likely matching frames; and
providing a sequence correlation in time using differences in frame numbers between pairs of the likely matching frames to identify a likely matching video sequence.
21. The method of claim 20 further comprising:
calculating a unique information value of signatures and selecting signatures having high unique information content that retain differentiating information between video sequences or items of interest.
22. The method of claim 20 wherein the correlation is performed in space using a location and scale of the matching feature points or objects of pairs of matching original and query signatures.
23. A method of generating a likelihood score for matching frames of a query video and an original video, the method comprising:
generating a similarity measure between a query video and an original video based on individual frame similarity scores that exceed a threshold score, wherein the original video is an entry in a video database;
generating a time correlation using relative differences in frame numbers of the original video and the query video; and
generating a correlation score between the original video and the query video by using a combination of the similarity measure and the time correlation.
24. The method of 23 where the time correlation uses a relative time rate between the original video and the query video as determined by a first few matching signatures between the original video and the query video.
25. The method of claim 23 wherein the correlation score between the query video and original video includes a factor of a change in the signatures of the each sequence in the query or that includes during the signature score computation a correlation that emphasizes the matches between changing information of the query signature.
26. A method to convert documents or activity such as online user session information or any natural event or activity into multi-dimensional vectors, the method comprising:
classifying documents, events, activity for learning by inference by a multi-dimensional vector; and
expecting certain behavior or next state in an activity, wherein the expected next state or the certain behavior is generated by a decision tree or a rule based system that takes as input one or more identified documents or classifications.
27. The method of 26 wherein identification of a video or object is used along with other user and online website parameters to predict a user state or a user activity interest.