Efficient content based video retrieval
Various disclosed embodiments relate to video content analysis based in part upon the detection of shot transitions. In some embodiments, a process and computer system for detecting shot transitions in a video is used to separate a video sequence into a series of “shots” having multiple frames. These shots may then be used for additional processing, e.g., content detection within the video frames.
1. A computer-implemented method for identifying a shot transition in an image sequence, comprising:
determining a first ordering of a first multiple of blocks in a first frame of the image sequence based upon block values associated with the first multiple of blocks;
determining a second ordering of a second multiple of blocks in a second frame of the image sequence based upon block values associated with the second multiple of blocks;
determining a distance between the first frame and a second frame based upon a permutation of the first ordering of the first multiple of blocks so as to correspond to the second ordering of the second multiple of blocks;
storing the distance in a time series of past frame comparisons, the past frame comparisons including distances between frames that precede the first and second frame;
determining the existence of a shot transition based upon distances in at least a portion of the time series of past frame comparisons;
identifying a first shot and a second shot based on the determined shot transition, wherein the first shot is comprised of a plurality of frames that precede the shot transition and the second shot is comprised of a plurality of frames that follow the shot transition;
processing the first shot with a first video tool adapted for video conditions of the first shot; and
processing the second shot with a second video tool adapted for video conditions of the second shot.
2. The computer-implemented method of claim 1 , wherein determining the existence of a shot transition comprises determining a pattern in the time series.
3. The computer-implemented method of claim 2 , further comprising:
determining a midpoint of the shot transition based upon the pattern in the time series.
4. The computer-implemented method of claim 1 , wherein determining a distance comprises determining a number of reorderings between the first multiple of blocks in the first frame and the second multiple of blocks in the second frame.
5. The computer-implemented method of claim 1 , wherein determining a distance between the first frame and a second frame based upon the permutation comprises:
identifying a number of position relocations in the first ordering of the first multiple of blocks in the first frame needed to create the block ordering in the second ordering of the second multiple of blocks in the second frame.
6. The computer-implemented method of claim 1 , further comprising:
grayscaling the first frame and the second frame before determining the block values of the first multiple of blocks and the block values of the second multiple of blocks, and wherein the block values comprise an average of the grayscale values within each block.
7. The computer-implemented method of claim 1 , wherein the first frame and the second frame comprise one of I-frames, P-Frames, or B-frames.
8. A non-transitory computer-readable medium comprising instructions executable by at least one processor to cause one or more computer systems to:
determine a first ordering of a first multiple of blocks in a first frame of the image sequence based upon block values associated with the first multiple of blocks;
determine a second ordering of a second multiple of blocks in a second frame of the image sequence based upon block values associated with the second multiple of blocks;
determine a distance between the first frame and a second frame based upon a permutation of the first ordering of the first multiple of blocks so as to correspond to the second ordering of the second multiple of blocks;
store the distance in a time series of past frame comparisons, the past frame comparisons including distances between frames that precede the first and second frame;
determine the existence of a shot transition based upon distances in at least a portion of the time series of past frame comparisons;
identify a first shot and a second shot based on the determined shot transition, wherein the first shot is comprised of a plurality of frames that precede the shot transition and the second shot is comprised of a plurality of frames that follow the shot transition;
process the first shot with a first video tool adapted for video conditions of the first shot; and
process the second shot with a second video tool adapted for video conditions of the second shot.
9. The non-transitory computer-readable medium of claim 8 , wherein determining the existence of a shot transition comprises determining a pattern in the time series.
10. The non-transitory computer-readable medium of claim 9 , further comprising:
determining a midpoint of the shot transition based upon the pattern in the time series.
11. The non-transitory computer-readable medium of claim 8 ,
wherein determining a distance comprises determining a number of reorderings between the first multiple of blocks in the first frame and the second multiple of blocks in the second frame.
12. The non-transitory computer-readable medium of claim 8 , wherein determining a distance between the first frame and a second frame based upon the permutation comprises:
identifying a number of position relocations in the first ordering of the first multiple of blocks in the first frame needed to create the block ordering in the second ordering of the second multiple of blocks in the second frame.
13. The non-transitory computer-readable medium of claim 8 , wherein the instructions are further executable by at least one processor to cause one or more computer systems to:
grayscale the first frame and the second frame before determining the block values of the first multiple of blocks and the block values of the second multiple of blocks, and wherein the block values comprise an average of the grayscale values within each block.
14. The non-transitory computer-readable medium of claim 8 , wherein the first frame and the second frame comprise one of I-frames, P-Frames, or B-frames.
15. A computer system comprising:
at least one processor;
a memory comprising instructions executable by the at least one processor to cause the computer system to:
determine a first ordering of a first multiple of blocks in a first frame of the image sequence based upon block values associated with the first multiple of blocks;
determine a second ordering of a second multiple of blocks in a second frame of the image sequence based upon block values associated with the second multiple of blocks;
determine a distance between the first frame and a second frame based upon a permutation of the first ordering of the first multiple of blocks so as to correspond to the second ordering of the second multiple of blocks;
store the distance in a time series of past frame comparisons, the past frame comparisons including distances between frames that precede the first and second frame;
determine the existence of a shot transition based upon distances in at least a portion of the time series of past frame comparisons;
identify a first shot and a second shot based on the determined shot transition, wherein the first shot is comprised of a plurality of frames that precede the shot transition and the second shot is comprised of a plurality of frames that follow the shot transition;
process the first shot with a first video tool adapted for video conditions of the first shot; and
process the second shot with a second video tool adapted for video conditions of the second shot.
16. The computer system of claim 15 , wherein determining the existence of a shot transition comprises determining a pattern in the time series.
17. The computer system of claim 15 , the memory further comprising instructions executable by the at least one processor to determine a midpoint of the shot transition based upon the pattern in the time series.
18. The computer system of claim 15 , wherein determining a distance comprises determining a number of reorderings between the first multiple of blocks in the first frame and the second multiple of blocks in the second frame.
19. The computer system of claim 15 , wherein determining a distance between the first frame and a second frame based upon the permutation comprises:
identifying a number of position relocations in the first ordering of the first multiple of blocks in the first frame needed to create the block ordering in the second ordering of the second multiple of blocks in the second frame.
20. The computer system of claim 15 , the memory further comprising instructions executable by the at least one processor to:
grayscale the first frame and the second frame before determining the block values of the first multiple of blocks and the block values of the second multiple of blocks, and wherein the block values comprise an average of the grayscale values within each block.
21. The computer system of claim 15 , wherein the first frame and the second frame comprise one of I-frames, P-Frames, or B-frames.