Text detection in videos
Systems and methods for detecting text in videos. To address problems with conventional Optical Character Recognition (OCR) systems, the present disclosure provides detection of text for improved OCR. Aspects of the present disclosure can, therefore, be utilized to detect a textual logo in videos, including when the text of the textual logo is clearly visible and when the text is inferred. Thus, examples capture appearance time of a textual logo from a video view perspective. Aspects use a multi-threshold pipeline for detecting video frames including the textual logo. A textual-visual scoring system is additionally used to leverage visual aspects of text in logos. A shot detection system is used to detect inferred text beyond a detected video frame. One or more verification models can be further applied.
1. A computer-implemented method, comprising:
receiving predicted text derived from processing frames in a video via optical character recognition (OCR);
determining a visual distance of the predicted text to target text;
determining, within a shot, a set of detected frames that include the target text by applying a first distance threshold to the visual distance of the predicted text;
extending the set of detected frames within the shot by applying a second distance threshold to the visual distance of the predicted text within the shot;
determining a sequence of frames in the shot that includes the extended set of detected frames; and
outputting the sequence of frames as results including the target text.
2. The method of claim 1 , wherein extending the set of detected frames further comprises determining a frame in the set of detected frames that has a visual distance corresponding to a representative prediction, wherein the frame is included in the shot.
3. The method of claim 1 , wherein extending the set of detected frames further comprises:
determining boundaries of the detected frames; and
extending the boundaries to a beginning of the shot from an earliest detected frame and from a latest detected frame to an end of the shot.
4. The method of claim 1 , wherein the first distance threshold is stricter than the second distance threshold.
5. The method of claim 1 , wherein determining the visual distance comprises using optimal transport to visually compare characters in the predicted text to the target text.
6. The method of claim 1 , further comprising verifying the sequence of frames prior to outputting the sequence of frames.
7. The method of claim 6 , wherein verifying the sequence of frames comprises:
determining the predicted text is a frequently occurring word, wherein the target text is a textual logo; and
applying a weight to the visual distance to penalize the predicted text.
8. A system, comprising:
a processing system; and
memory storing instructions that, when executed by the processing system, cause the system to:
receive predicted text derived from processing frames in a video via optical character recognition (OCR);
determine a visual distance of the predicted text to target text;
determine, within a shot, a set of detected frames that include the target text by applying a first distance threshold to the visual distance of the predicted text;
extend the set of detected frames within a shot by applying a second distance threshold to the visual distance of the predicted text within the shot;
determine a sequence of frames in the shot that includes the extended set of detected frames; and
output the sequence of frames as results including the target text.
9. The system of claim 8 , wherein extending the set of detected frames comprises determining a frame in the set of detected frames has a visual distance corresponding to a representative prediction, wherein the frame is included in the shot.
10. The system of claim 8 , wherein extending the set of detected frames comprises:
determining boundaries of the detected frames; and
extending the boundaries to a beginning of the shot from a earliest detected frame and from a latest detected frame to an end of the shot.
11. The system of claim 8 , wherein the first distance threshold is stricter than the second distance threshold.
12. The system of claim 8 , wherein determining the visual distance comprises using optimal transport to visually compare characters in the predicted text to the target text.
13. The system of claim 8 , wherein the instructions cause the system to verify the sequence of frames prior to outputting the sequence of frames.
14. The system of claim 13 , wherein:
the target text is a textual logo; and
verifying the sequence of frames comprises:
determining the predicted text is a frequently occurring word; and
applying a weight to the visual distance to penalize the predicted text.
15. A computer readable media comprising instructions, which when executed by a computer, cause the computer to:
receive a plurality of predicted texts derived from processing frames in a video via optical character recognition (OCR);
determine a visual distance of each predicted text of the plurality of predicted texts to target text;
determine a set of detected frames that include the target text by applying a first distance threshold to the visual distance of each predicted text;
extend the set of detected frames within a shot by applying a second distance threshold to the visual distance of each predicted text within the shot, wherein the second distance threshold is less strict than the first distance threshold;
determine a sequence of frames in the shot that includes the extended set of detected frames; and
output the sequence of frames as results including the target text.
16. The computer readable media of claim 15 , wherein extending the set of detected frames comprises determining at least one frame in the set of detected frames has a visual distance corresponding to a representative prediction, wherein the at least one frame is included in the shot.
17. The computer readable media of claim 15 , wherein in extending the set of detected frames, the instructions further cause the computer to:
determine boundaries of the detected frames; and
extend the boundaries to a beginning of the shot from a earliest detected frame and from a latest detected frame to an end of the shot.
18. The computer readable media of claim 15 , wherein in determining the visual distance, the instructions cause the computer to use optimal transport to visually compare characters in the plurality of predicted texts to the target text.
19. The computer readable media of claim 15 , wherein the instructions cause the computer to verify the sequence of frames prior to outputting the sequence of frames.
20. The computer readable media of claim 15 , wherein:
the target text is a textual logo; and
verifying the sequence of frames comprises, for each predicted text in the sequence of frames:
determining the predicted text is a frequently occurring word; and
applying a weight to the visual distance to penalize the predicted text.