Detecting sentinel frames in video delivery using a pattern analysis
Particular embodiments extract features for frames from a video. Then, a sequence of frames is identified based on a pattern analysis of frames based on the features. This pattern may be used to select sentinel features from the sequence of frames. For example, a pattern may include a first sentinel sequence followed by a transitional frame and then a second sentinel sequence. The transitional frame may be a black frame that is used to identify back-to-back sentinel sequences. Then, a sentinel frame that demarks a transition from a first content type to a second content type is identified. For example, the frames on either side of the black frame may be very similar and be identified as sentinel frames that mark a transition from program content to advertisement content. The above process may allow automated detection of sentinel frames that can run without user supervision.
1. A method comprising:
extracting, by a computing device, frame features for a plurality of frames from a video;
identifying, by the computing device, a pattern from a sequence of frames of the plurality of frames, the pattern being identified based on a pattern analysis using the frame features for frames in the sequence of frames;
clustering, by the computing device, a set of candidate frames into one or more groups based on the frame features for the set of candidate frames;
selecting, by the computing device, sentinel features for each of the one or more groups from the frame features for frames in the sequence of frames based on the pattern;
generating a sentinel frame for each of the one or more groups, using the sentinel features for the frames in each of the one or more groups;
outputting, by the computing device, a set of sentinel frames for each of the one or more groups using the sentinel features, the set of sentinel frames identifying a transition in the video from a first content type to a second content type, wherein the sentinel frame includes the sentinel features;
storing, by the computing device, the set of sentinel frames in a sentinel database;
analyzing, by the computing device, a new video, the analyzing includes repeating the extracting, identifying, clustering, and generating steps;
comparing, by the computing device, the generated sentinel frames for the new video to the sentinel frames stored in the sentinel database;
determining, by the computing device, the generated sentinel frames for the new video includes at least one different sentinel frame from the sentinel frames stored in the sentinel database; and
updating, by the computing device, the sentinel database based on the at least one different sentinel frame.
2. The method of claim 1 , wherein identifying the pattern comprises:
selecting a transitional frame in the sequence of frames, wherein the transitional frame is different from the sentinel frame; and
using at least one frame in the sequence of frames that is before or after the transitional frame to select the sentinel features.
3. The method of claim 2 , wherein selecting the sentinel features comprises using a frame in the sequence of frames on both sides of the transitional frame in the sequence of frames to select the sentinel features.
4. The method of claim 2 , wherein identifying the pattern comprises:
comparing a first frame on a first side of the transitional frame to a second frame on a second side of the transitional frame to determine if the first frame and the second frame are similar within a threshold; and
when the first frame and the second frame are similar within the threshold, identifying the sequence of frames as including the pattern.
5. The method of claim 2 , further comprising:
grouping sequential transitional frames in the sequence of frames into a group, wherein the at least one frame in the sequence of frames is near the group within a threshold.
6. The method of claim 2 , wherein the transitional frame is a solid color frame.
7. The method of claim 1 , wherein the pattern comprises a first sequence of frames and a transitional frame.
8. The method of claim 7 , further comprising:
locating a second sequence of frames in addition to the first sequence of frames and the transitional frame, wherein:
a first frame in the first sequence of frames matches a second frame in the second sequence of frames within a threshold, and
the sentinel features are selected from the first frame or the second frame.
9. The method of claim 1 , wherein the sentinel features comprise features determined from decoding the frames of the video.
10. The method of claim 1 , further comprising:
receiving an input identifying the pattern.
11. The method of claim 1 , wherein the set of sentinel frames are not previously known before selecting the sentinel features.
12. An apparatus comprising:
one or more computer processors; and
a non-transitory computer-readable storage medium comprising instructions that, when executed, control the one or more computer processors to be configured for:
extracting frame features for a plurality of frames from a video;
identifying a pattern from a sequence of frames of the plurality of frames, the pattern being identified based on a pattern analysis using the frame features for frames in the sequence of frames;
clustering a set of candidate frames into one or more groups based on the frame features for the set of candidate frames;
selecting sentinel features for each of the one or more groups from the frame features for frames in the sequence of frames based on the pattern;
generating a sentinel frame for each of the one or more groups, using the sentinel features for the frames in each of the one or more groups;
outputting a set of sentinel frames for each of the one or more groups using the sentinel features, the set of sentinel frames identifying a transition in the video from a first content type to a second content type, wherein the sentinel frame includes the sentinel features;
storing the set of sentinel frames in a sentinel database;
analyzing, by the computing device, a new video, the analyzing includes repeating the extracting, identifying, clustering, and generating steps;
comparing, by the computing device, the generated sentinel frames for the new video to the sentinel frames stored in the sentinel database;
determining, by the computing device, the generated sentinel frames for the new video includes at least one different sentinel frame from the sentinel frames stored in the sentinel database; and
updating, by the computing device, the sentinel database based on the at least one different sentinel frame.
13. The method of claim 1 , wherein the frame features extracted include one or more of: a color layout descriptor, an edge histogram, on-screen textual markings, on-screen logos, and ticker information.
14. The method of claim 1 , wherein the first content type is program content and the second content type is advertising content.