System and method for extracting text captions from video and generating video summaries
View Patent ↗Caption boxes which are embedded in video content can be located and the text within the caption boxes decoded. Real time processing is enhanced by locating caption box regions in the compressed video domain and performing pixel based processing operations within the region of the video frame in which a caption box is located. The captions boxes are further refined by identifying word regions within the caption boxes and then applying character and word recognition processing to the identified word regions. Domain based models are used to improve text recognition results. The extracted caption box text can be used to detect events of interest in the video content and a semantic model applied to extract a segment of video of the event of interest.
1. A method of decoding a caption box in video content comprising:
determining at least one expected location of a caption box in a frame of the video content;
determining at least one caption box mask within the expected location;
identifying frames in the video content as caption frames if the current frame exhibits substantial correlation to the at least one caption box mask within the expected caption box location;
for at least a portion of the caption frames, identifying word regions within the confines of the expected location; for each word region,
identifying text characters within the region; and
processing the identified text characters.
2. The method of decoding a caption box according to claim 1 further comprising:
comparing the text characters in the word region against a domain specific model to enhance word recognition,
determining a word region type for at least one of the identified word regions;
wherein the video content is of a sporting event and wherein the word region types are selected from the group consisting of data points that indicate the current state of the event.
3. The method of decoding a caption box according to claim 2 wherein the data point is a period.
4. The method of decoding a caption box according to claim 2 wherein the data point is a quarter.
5. The method of decoding a caption box according to claim 2 wherein the data point is field position.
6. The method of decoding a caption box according to claim 2 wherein the data point is the number of shots on goal.
7. A non-transitory computer-readable storage medium storing a program for causing a computer to implement a method of decoding a caption box in video content comprising:
determining at least one expected location of a caption box in a frame of the video content;
determining at least one caption box mask within the expected location;
identifying frames in the video content as caption frames if the current frame exhibits substantial correlation to the at least one caption box mask within the expected caption box location;
for at least a portion of the caption frames, identifying word regions within the confines of the expected location;
for each word region, identifying text characters within the region;
and processing the identified text characters.
8. The non-transitory computer-readable storage medium of claim 7 wherein the method includes the step of comparing the text characters in the word region against a domain specific model to enhance word recognition.
9. A system for decoding a caption box in video content comprising:
location means for determining at least one expected location of a caption box in a frame of the video content;
determining means, coupled to the location means and receiving the at least one expected location therefrom, for determining at least one caption box mask within the expected location;
frame identifying means, coupled to the determining means and location means and receiving the at least one caption box mask and the at least one expected location therefrom, for identifying frames in the video content as caption frames if the current frame exhibits substantial correlation to the at least one caption box mask within the expected caption box location;
word region identifying means, coupled to the frame identifying means and receiving the identified caption frames therefrom, for at least a portion of the caption frames, identifying word regions within the confines of the expected location;
text character means, coupled to the word region identifying means and receiving the word regions therefrom, for each word region, identifying text characters within the region; and processing the identified text characters.
10. The system of claim 9 wherein the system includes a comparing means, coupled to the text character means and receiving the text characters therefrom, for comparing the text characters in the word region against a domain specific model to enhance word recognition.
11. The system of claim 9 wherein the location means includes a features means for evaluating motion features of the video frame in the compressed domain, evaluating texture features of the video frame in the compressed domain, and identifying regions having low motion features and high texture features as candidate caption box regions.
12. The system of claim 10 wherein the location means includes a features means for evaluating motion features of the video frame in the compressed domain, evaluating texture features of the video frame in the compressed domain, and identifying regions having low motion features and high texture features as candidate caption box regions.
13. The system of claim 9 wherein the system includes a removal means, coupled to the frame identifying means, the determining means and the location means and receiving the identified caption frames, the at least one caption box mask and the at least one expected location therefrom, for evaluating the identified caption frames, within the caption box location, for changes in content and removing caption frames from word region processing which do not exhibit a change in content.
14. The system of claim 9 wherein the system includes an interval means, coupled to the frame identifying means and receiving the identified caption frames therefrom, for selecting a subset of caption frames based on a predetermined time interval and sending that subset to the word region identifying means.
15. The system of claim 9 wherein the system includes a number means, coupled to the frame identifying means and receiving the identified caption frames therefrom, for determining a subset of caption frames by selecting caption frames based on a predetermined number of intervening caption frames and sending that subset to the word region identifying means.